Learning to Improve:

A First Look at Trends and Bright Spots in School Systems Performance

Post-Pandemic

State of Missouri

January 2023

Analysis and visualization by

PAGE 2

Welcome & Agenda

1. State and Regional Education Data Analysis

​

• Insights from Fast-Improving Systems

​

• A New Public Data Tool and Improvement Resource

Meet Steve Cartwright

PAGE 3

Steve Cartwright

Founder, Exponent Education

• Early research experience at Urban Institute and DC Public Schools

​

• Served as Fellow, Strategic Data Project, Harvard Center for Education Policy Research

​

• Analytics consultant with schools, systems, and support orgs across the country

Today’s content

PAGE 4

1

Pandemic impact on national and state achievement

2

New evidence on regional achievement

3

Finding and learning from outliers

The Covid-19 pandemic significantly impacted student academic achievement across the country

PAGE 5

• Losses in these key milestones ranged from about half to nearly all of the gains made between 2000 and

2019.

272

276

278

280

282

283

284

281

282

281

273

211

216

217

220 220

220

221

221

221

219

216

2000 2003 2005 2007 2009 2011 2013 2015 2017 2019 2022

National Assessment of Educational Progress Composite Score, 2009-2022, All Students

Grade 8 Proficient

-8 points

(loss of 86% of 2000 to 2019 gains)

-3 points

(loss of 41% of 2000 to 2019 gains)

Grade 8 Math

Grade 4 Reading

No Child Left Behind: national emphasis on assessment and accountability

Race to the Top: national emphasis on standards and teacher evaluation

Current era interrupted by global pandemic

• Average achievement increased significantly between 2000 and 2013 before stagnating and declining.

• The pandemic most affected students’ math performance, but may have lingering effects in reading.

Information on NAEP cut scores for Basic and Proficient performance can be found here.

Grade 8 Basic

Grade 4 Basic

Grade 4 Proficient

In Missouri, average declines were similar to the nation, but losses for low-income students were particularly large

PAGE 6

• Low-income students in Missouri were disproportionately affected and experienced some of the nation’s largest declines.
• Low-income students in Missouri lost more ground than their peers in most other states.

222

221

221

224

220

222

223 223

218

213

208

209

208

210

207

211

211

211

207

197

279

276

281

286

282

283

²⁸¹ 281 ²⁸¹

272

263

262

266

272

269

271

266

268

267

256

National Assessment of Educational Progress - Missouri Composite Score, 2009-2022

-10 points (loss of >1100%

of 2003 to 2019 loss)

2003 2005 2007 2009 2011 2013 2015 2017 2019 2022

All Students

-9 points (loss of >400% of

2003 to 2019 gains)

Low-Income Students

-11 points (loss of >250% of

2003 to 2019 gains)

All Students

-5 points (loss of ~110%

greater than 2003 to 2019 loss)

Low-Income Students

Grade 8 Math

Grade 4 Reading

Missouri’s low-income students declined more than their peers in most states and now perform near the bottom nationally

PAGE 7

Grade 4 Reading Composite Score

NAEP State Rankings for Low-Income Students

Grade 8 Math Composite Score

NAEP State Rankings for Low-Income Students

Today’s content

PAGE 8

1

Pandemic impact on national and state achievement

2

New evidence on regional achievement

3

Finding and learning from outliers

Districts’ results are aggregated into one of five mutually exclusive groupings

PAGE 9

• Urban districts: KCPS and SLPS

​

• Urban charter:

Charter LEAs operating in Kansas City and St. Louis City

​

• Suburban counties:

“Charter” counties of Clay, St. Charles, St. Louis, Jefferson, and Jackson (minus KCPS)

​

• Semi-urban counties:

The 13 “first class” counties like Cole, Cape Girardeau, etc.

​

• Rural counties:

All others

ELA performance continued to decline from 2021 to 2022 statewide and remained relatively flat locally

PAGE 10

• Statewide and in most geographies, ELA proficiency continued to decline from ‘21 to ‘22.
• Both urban districts and charters improved modestly (+1% and +2% points, respectively).
• No region or sector has reached pre-pandemic levels of performance.

+1

+2

Interact with these data on www.theopportunitytrust.org

Math proficiency shows some signs of “rebounding” statewide, unlike ELA

PAGE 11

• Mirroring national trends, students lost more ground in mathematics than reading.
• Some signs of ”rebounding” exist in math: students in every geography and sector are improving to varying degrees, through no sector is back to pre-pandemic achievement.

+4

+5

+8

+6

+3

+3

Interact with these data on www.theopportunitytrust.org

Breaking down the urban results highlights some differences across sectors

PAGE 12

+1

+2

+3

-1

+6

+3

+9

+7

• In ELA, gains were modest in both St. Louis and Kansas City, with KCPS declining slightly.
• In math, the charter sector rebounded at 2x the rate of district schools in St. Louis and 1.3x the rate of district schools in Kansas City

Twenty years of research identifies key milestones in children's development that predict meaningful life outcomes

PAGE 13

Kindergarten readiness

3^rd grade reading proficiency

8^th grade math proficiency

4^th grade math proficiency

A local study found that students entering kindergarten behind their peers continued to lag 18 months later (University of Missouri 2019). Another study from Ohio found nearly three in four (74%) students who tested positively for a potential problem on a kindergarten readiness assessment scored below the passing promotion score on the third grade ELA assessment (Justice et al 2019).

Students who were not proficient in third grade reading are four times as likely not to graduate from high school as those who were reading proficiently in third grade. For children who lived in a high- poverty neighborhood for at least a year and were not reading proficiently in third grade, they are six times as likely not to graduate (Hernandez 2012).

Researchers studying ten-year- old students’ mathematical knowledge in the US and the UK found that their mastery of fractions was highly predictive of their overall mathematical and algebraic knowledge in high school, five to six years after initial testing (Siegler et al 2012).

Researchers consider eighth grade math the “gatekeeper” to higher math and science courses and other postsecondary opportunities for both college and career track students (Wang and Goldsmith 2003).

Milestone: 3^rd grade reading proficiency

PAGE 14

• The 3^rd grade results mirror the overall trend of stagnation statewide with some modest improvements locally.
• In Missouri’s cities, more than 50% of 3^rd grade students score at the lowest level of performance.

​

• Nearly two thirds (63%) of urban district students scored below basic - the lowest of four levels of performance.

Interact with these data on www.theopportunitytrust.org

Milestone: 8^th grade math proficiency

PAGE 15

Interact with these data on www.theopportunitytrust.org

• “Rebounding” in math is shown not just in proficiency improvements, but also in reductions in the percentage of students below basic statewide.
• Urban districts and charters significantly reduced the percentage of below basic students.

Today’s content

PAGE 16

1

Pandemic impact on national and state achievement

2

New evidence on regional achievement

3

Finding and learning from outliers

The schools best helping students recover aren’t just those increasing proficiency rates

PAGE 17

Interact with these data on www.theopportunitytrust.org

• Proficiency rates are crude measures to use when evaluating change over time.
• Movement across all four performance levels provides a more detailed view of change.

​

​

​

48%

​

​

40%

​

26%

​

​

42%

19%

14%

7%

4%

2022

Proficient

Basic

2021

Below Basic

Advanced

​

​

​

43%

​

​

​

42%

​

​

28%

​

27%

21%

21%

8%

10%

2022

Proficient

Basic

2021

Below Basic

Advanced

ELA Performance by Level, 2021 to 2022 Low-Income Students

Missouri

Brookside Charter (KC)

No change

-2% pts.

+1% pt.

+1% pt.

+5% pts.

+3% pts.

+8% pts.

-16% pts.

• Unlike the state, Brookside Charter increased proficiency rates and decreased the rate of students below basic.

Fast-recovering schools/systems for low-income students include a mix of districts and charters

PAGE 18

• Outlier schools achieve outsized performance index changes with low-income students (and serve a majority of low-income students overall).

School composition at least 50% Low Income

Compute a performance index to capture movement across all levels of performance

Interact with these data on www.theopportunitytrust.org Note: Only schools with <10% of students omitted are shown.

Outliers are improving significantly faster than district and county averages

PAGE 19

Interact with these data on www.theopportunitytrust.org Note: Only schools with <10% of students omitted are shown.

Rural Counties

Brookside Charter

Zalma R-V

+29.6 index points

+27.2 index points

Missouri

-4.4 index points

-2.0 index points

-5.2 index points

Kansas City 33

Outliers present an example of what’s possible when the right conditions for improvement are in place

PAGE 20

29

25

80

64

70

60

50

40

30

20

10

0

ELA

Math

2022 Actual

100

90

80

2027 Projection

Missouri Low-Income Student Proficiency Rate Projection

• Current proficiency rates for low-income students are 29% in ELA and 25% in math.

​

• If every school improved their proficiency rates for low- income students at Zalma’s rate* for the next five years, statewide low-income student achievement would dramatically improve.

* +10.2% points in ELA and +7.9% points in math.

**See Chetty, R., Friedman, J. N., & Rockoff, J. (2011). New evidence on the long-term impacts of tax credits. IRS white paper.

In an average class of 30 students, this amounts to 3 additional students proficient in ELA and 2 additional students proficient in math per year.

• These increased test scores translate into significant improvements in college attendance and early career earnings.**

Statewide all student a

​

​

verage

43

39

Necessary Conditions for Improvement

PAGE 21