1 of 74

Measuring Growth

�What growth is (and isn’t) �How to estimate growth correctly (and incorrectly)�What’s wrong with the broken CDE Dashboard

�by �Steve Rees�K12 Measures��For the �California Charter School Association

March 20, 2024

2 of 74

Measuring Growth of Learning �of a Single Student

3 of 74

Not as easy as measuring height. But we are aiming for something as clear, and just as easy to explain.

4 of 74

Designed to measure growth

Scale enables estimating growth within a year and across the grade spans

Given 3x/year in reading, language usage and math

Delivers 45-60 questions, untimed

Norms for scale score and growth

Using the NWEA Measures of Academic Progress

5 of 74

Individual Student Growth

Morgan from grade 4 to start of grade 7

math

Here are Morgan’s math results from middle of grade 4 to the start of grade 7. His scores are connected by the black line at the top. Each hollow dot connected by that line is a test event. The last two solid dots are projected scores……. Below Morgan’s score is a dotted line which represents average achievement for all kids tested nationally. Below that is a dash-dot line which represents the district’s students’ average scale score. In the background you see five colors. Each color represents how one-fifth of students scored nationally – a range of scores for each quintile. This enables us to conclude that although Morgan’s scores wandered up and down, then up and down again, most of the time, his scores were in the 2^nd quintile – between the 60^th and 80^th percentile of students nationally.

6 of 74

Individual Student Growth

Morgan from grade 4 to start of grade 7

reading

7 of 74

Individual Student Growth

Morgan from grade 4 to start of grade 7

math

reading

8 of 74

Analytic Exercise

What does a multi-year view of test scores reveal that a one-year view does not?

Guiding questions

9 of 74

Analytic Exercise

What does a multi-year view of test scores reveal that a one-year view does not?

What more do you know by seeing results for both math and reading together?

Guiding questions

10 of 74

Individual Student Growth

Connor from grade 4 to start of grade 7

math

reading

11 of 74

Individual Student Growth

Leilani from grade 4 to start of grade 7

math

reading

Here’s Leilani, also from grade 4 to start of grade 7. Interestingly, her math scores are also characterized by ups of a dramatic sort (big jumps), and downs of a moderate sort. She’s made terrific progress in 6^th grade, moving up to the bottom of the middle quintile (yellow). Note that NWEA’s MAP predicts that her next two test events will be modest gains up, about the same amount of gain as the national average. I don’t believe that’s likely at all. Based on her pattern, and the patterns of the prior students, I have a hunch that big gains will be followed by declines. What do you think?....... Her reading scores reflect a learning rate that’s higher than that of the national or district average. I’m commenting on the slope of the line. But note she still had 2 test events where her scale scores declined a bit.

12 of 74

Growth is not linear. Expect ups and downs.

Some students have extended periods of flat results, followed by bursts of growth.

Sometimes, gains in scores occur over summer, when no schooling has occurred.

Math and reading patterns often differ.

Observations of Learning Growth Patterns

13 of 74

Measuring Growth of Learning �of Grad Class Cohorts Over Time

14 of 74

Planning with evidence from NWEA MAP results

15 of 74

Planning with evidence from NWEA MAP results

Our 3^rd graders’ reading scores improved a lot in just 15 weeks, from about the 24^th to about the 38^th percentile.

16 of 74

NWEA MAP contains assumptions you need to know.

Entity

Students (individual)

Subgroup

Classroom

Grade level

School or�district

Graduating class cohort

Metric

Scale score

Distance�from�standard

Percentage of students meeting or exceeding standard

Percentile

Time

<1 year

1 year

2 years

3 years

4+ years

Context

Your school alone

Your district

Your county average

Similar schools

All schools

State average

Norms

Vantage

Point

Cross-�sectional

Quasi-�longitudinal

Longitudinal

17 of 74

Designed to measure growth using CAASPP

Looking at more or less same kids over 3 to 8 years

Comparing results to schools with highly similar students

Norms for scale score and growth

Using the K12 Measures Assessment Explorer

18 of 74

What is a school’s effect on what students know and can do.

The question we’re asking drives the evidence we’re building

19 of 74

Growth at School Level

What is the question we are trying to answer? It is too often this question. Not good.

“Did our kids in grades 3-5 make as much progress in math last year as kids in the same grade level in California?”

20 of 74

Analysis of same kids over longer time enables us to reduce the noise of student variability and see the school’s effect.

“Did our kids in the graduating classes of 2027 and 2028 make as much progress in math as California kids over the three years they’ve taken the CAASPP?”

21 of 74

Adding a context of schools with highly similar students, enables you to make claims like this.

“Over the last 3 years, our kids in the graduating classes of 2027 and 2028 made more progress in math than highly similar kids in 12 of 15 schools a lot like ours.”

22 of 74

Elements of similarity

Students

Parent education

Free or reduced-price lunch

English language fluency

23 of 74

Elements of student similarity

24 of 74

Elements of student similarity

25 of 74

Elements of student similarity

26 of 74

Decide whose growth to measure

View that comparison from a certain vantage point

Select the right metric (scale score)

Choose a period of time

Decide who to compare to whom

To estimate a school’s effect on students, we have to …

27 of 74

Restructured results by graduating class cohorts

Used scale scores

Viewed same students (more or less)

Over as many years as possible

Compared to highly similar students in schools serving same grade range

To build growth estimates from CAASPP results …

28 of 74

The Assumptions of the �K12 Measures Assessment Explorer

Entity

Students (individual)

Subgroup

Classroom

Grade level

School or�district

Graduating class cohort

Metric

Scale score

Distance�from�standard

Percentage of students meeting or exceeding standard

Percentile

Time

<1 year

1 year

2 years

3 years

4+ years

Context

Your school alone

Your district

Your county average

Similar schools

All schools

State average

Norms

Vantage

Point

Cross-�sectional

Quasi-�longitudinal

Longitudinal

29 of 74

CAASPP reporting site looks at grad class cohorts

Source: CAASPP reporting site

30 of 74

The Case of Napa Valley USD’s Middle School Math Sag: Did COVID Cause It?

Napa Valley USD

About 16,500 students
Math results in elementary schools contrast with results in middle schools
Can comparability shed light on the question: is it due to COVID?

31 of 74

Napa Valley USD’s Class of 2027

State average scale score

32 of 74

Napa Valley USD’s Class of 2027 in Context

33 of 74

Napa Valley USD’s Class of 2027 in Context

Gilroy USD

Napa Valley USD

34 of 74

Napa Valley USD’s Class of 2026 in Context

35 of 74

Napa Valley USD’s Class of 2028 in Context

This group of 1,100 Napa Valley kids in the grad class of 2028 were doing well in 3^rd grade in both ELA and math. Just below state avg scale score in ELA and just slightly below Gilroy. And in math as 3^rd graders in 2019 these kids were right at the state avg scale score. While sitting slightly under Gilroy’s students, they were still at the top of the stack of highly similar districts. ….. School was upended for them in grades 4 and 5. By the spring of grade 6 in 2022, in math Napa Valley’s students in this grad class were far below the state avg scale score, far below Gilroy’s students, and third from the bottom. By spring of the 7^th grade, their math results remained low….. Is there anything else I might do to improve the quality of evidence here? YES. Let’s restrict results to only those students who are English fluent.

36 of 74

Napa Valley USD’s Class of 2028 in Context

37 of 74

When evidence conflicts

The Dashboard versus K12 Measures and the�Stanford Educational Opportunity Project

Yuba River Charter School

189 students tested in 2023
Grass Valley, Nevada County
Waldorf model
Serving K-8 students
Launched in 1994

38 of 74

39 of 74

Designed to measure growth (learning rate)

National in scope, covering schools and districts

Looks at state tests from 2009-2018

Provides a context of socio-economic status

Stanford Educational Opportunity Explorer

40 of 74

Average Students’ Test Scores, 2009-2018

Average Students’ Test Scores, 2009-18

By Stanford Educational Opportunity Explorer

click here to see live version. Yuba River is #13 on the visualization

The Stanford team has averaged results over the ten years 2009 to 2018. They’ve also put results for ELA and math together in that average, largely to diminish the imprecision in the measures that result. Here you can see them where that red dot appears (added by me). The horizontal axis is a free/reduced lunch factor. To the left are schools with higher percentages of students getting free/reduced lunch, and to the right, fewer.

Yuba River is about in the middle of that.

The vertical axis indicates test scores, converted to grade level equivalents. The Stanford crew did this to speak the language of the public. Their students in grades K-8 had average test scores that put them about .67 of a grade level below the U.S. average. Grey dots in the background are schools that are outside of California. In blue you’ll see California schools with below avg test scores. In green you’ll see Calif schools with above avg test scores.

41 of 74

Average Students’ Learning Rates, 2009-18

By Stanford Educational Opportunity Explorer

click here to see live version. Yuba River is #13 on the visualization

42 of 74

By Stanford Educational Opportunity Explorer

Yuba River Charter School As Viewed by the Stanford Educational Opportunity Explorer 2009-2018

ELA and math results are combined to reach these conclusions.

43 of 74

Yuba River Charter School (Class of 2028) as �K12 Measures Assessment Explorer Sees It

Here’s the results for the class of 2028. There are 30 kids in this grad class cohort. I spoke with their principal, Denis Hill, and he explained to me that because they are a Waldorf school, they take a slower approach to teaching reading than other schools. Just introduce the students to letters and letter sounds in first grade, for example. And definitely no computers until 3^rd grade. As a result, the scores of their 3^rd graders are below the state avg scale score and close to the bottom of this group of 15 similar schools. But by grade 6 in math, they are well above the state avg scale score, and fifth highest in math….. By 7^th grade, these kids are 4th highest in math an even farther above state avg scale score. In ELA by 7^th grade, they are third highest and well above state avg scale score. Let’s look at another grad class cohort ….

……

44 of 74

Yuba River Charter School (Class of 2027) as �K12 Measures Assessment Explorer Sees It

45 of 74

Yuba River Charter School (Class of 2026) as �K12 Measures Assessment Explorer Sees It

46 of 74