1 of 69

NAEP Reading Frameworks

Dr. Laura Goadrich

Arkansas NAEP State Coordinator� laura.Goadrich@ade.arkanas.gov

https://tinyurl.com/mssht597

2 of 69

Outline

Overview of NAEP
NAEP Results & Assessment Schedule
NAEP Frameworks
International Assessments

2

3 of 69

What is NAEP?

The National Assessment of Educational Progress (NAEP) is the largest continuing and nationally representative assessment of what our nation’s students know and can do in core subjects.

An assessment designed to produce national, state, and select urban district level results.

NAEP results are for populations of students, not for individual students nor schools.
Measures student performance nationally and reports changes over time.
Allows comparisons between states and the nation.

3

4 of 69

4

NAEP School and Student Selection

Identify schools
Classify schools
Arrange schools
List schools
Select schools
Confirm schools

5 of 69

NAEP Administration

NAEP is administered over a six-week period.

Extended four weeks this year due to COVID
From January - March
Each student is only assessed in one subject area.
Students only take a small portion of the assessment.
Accommodations are provided as necessary for students with disabilities and English language learners. �

NAEP 2022 results will be available late October 2022

5

6 of 69

6

NAEP Sample Sizes

The results from the 2019 assessment are based on representative samples from 264 Arkansas schools.

Nation(public)= 50 states, DC, DoDEA (not PR)

Math	Nation (public)	Arkansas
Grade 4	143,600	2,300
Grade 8	142,200	2,300

Reading	Nation (public)	Arkansas
Grade 4	144,700	2,300
Grade 8	138,100	2,200

7 of 69

NAEP 2019 Results

The results of NAEP are released as �The Nation’s Report Card.

The report card provides national, state, and district-level results, results for different demographic groups, inclusion information, and sample questions.

NAEP Results are reported in two formats

Average Scaled Scores

Numeric scale
0 − 500 on mathematics and reading assessments
Scores cannot be compared across content areas

Achievement Levels

Categorical scale
NAEP Below Basic, NAEP Basic, NAEP Proficient, NAEP Advanced

7

8 of 69

NAEP Assessment Schedule

8

9 of 69

What is a NAEP framework?

NAEP frameworks are written for large-scale assessment, are curriculum neutral, and reflect aspirations for what all students should know and be able to do.
Each NAEP framework describes:

What is to be measured, �i.e., the content that will be assessed for the subject area;
How it will be measured, �e.g., the format of each NAEP assessment, the knowledge and skills to be assessed at grades 4, 8, and 12, sample items for each grade, weighting of the item pool, provision of ancillary materials, uses of technology, etc.;
How much content knowledge and skills are required for NAEP Basic, NAEP Proficient, and NAEP Advanced performance, �i.e., how much of the content domain students should know and be able to do for each NAEP achievement level.

9

https://www.nagb.gov/content/dam/nagb/en/documents/naep/NAEP-Frameworks-FAQ-FINAL.pdf

https://nces.ed.gov/nationsreportcard/frameworkhow.aspx

Consider NAEP frameworks as guidelines when you revise state or district curricula. NAEP frameworks present and explain what experts in a particular subject area consider important. Each framework contains a subject outline that describes what students at grades 4, 8, and 12 should know and be able to do in that NAEP content area. The frameworks also provide specific examples of challenging objectives and sample test questions to illustrate the content standards. This information may serve as a "road map" to help teachers and curriculum planners evaluate and revise current state or district standards.

Use the NAEP frameworks to evaluate how classroom instruction and assessment focus on each cognitive level. Frameworks frequently provide research-based information on important higher-order skills that students should master at each grade level. For example, by studying the mathematics framework teachers might see that most of their instruction in measurement addresses only procedural knowledge such as the ability to apply formulas for perimeter and area. In this manner, the NAEP framework may assist teachers in expanding their level of instruction to include more complex problem-solving exercises at a grade-appropriate level.

Use released NAEP constructed-response questions and their corresponding scoring guides, along with the frameworks, as models of how to use innovative, challenging assessment practices that measure a given framework skill. The sample scoring guides may help teachers improve the way they score student responses.

10 of 69

How to use the NAEP Frameworks?

Consider NAEP frameworks as guidelines when you revise state or district curricula. �

NAEP frameworks present and explain what experts in a particular subject area consider important.

Each framework contains a subject outline that describes what students at grades 4, 8, and 12 should know and be able to do in that NAEP content area.

The frameworks also provide specific examples of challenging objectives and sample test questions to illustrate the content standards.

This information may serve as a "road map" to help teachers and curriculum planners evaluate and revise current state or district standards.

11 of 69

NAEP Frameworks- 10 year development cycle

11

12 of 69

NAEP Reading Frameworks Updated

Current Reading Framework 2019
NAGB adopted the new Reading Framework for 2026)

12

13 of 69

NAEP Reading Framework

13

Grade	Literary	Informational
4	50%	50%
8	45%	55%
12	30%	70%

Conceptualizes reading as a dynamic cognitive process

Reading is an active and complex process that involves:

Understanding written text;
Developing and interpreting meaning;
Using meaning as appropriate to type of text, purpose, and situation.

This definition applies to the assessment of reading achievement on NAEP and is not intended to be an inclusive definition of reading or of reading instruction.

14 of 69

NAEP Reading Framework

14

15 of 69

NAEP Reading Framework

15

16 of 69

NAEP Reading Framework

16

17 of 69

NAEP Reading Framework

17

18 of 69

HOW OTHER STATES USE IT:

Created one page summary maps for each of the state-level resources: Math (gr 4, 8), Reading (gr 4, 8)
Math (Course of Study) and ELA created a 1-12 grade mapping between the NAEP framework and their state standards

ALABAMA

https://www.nationsreportcard.gov/ndecore/xplore/NDE

All students [view details]
Race/ethnicity used to report trends, school-reported [view details]
Race/ethnicity using 2011 guidelines, school-reported [view details]
Gender [view details]
National School Lunch Program eligibility, 3 categories [view details]
Disability status of student, including those with 504 plan [view details]
Disability status of student, excluding those with 504 plan [view details]
Status as English learner, 2 categories [view details]
Student disability or English learner status [view details]

The National Center for Education Statistics (NCES) has periodically published reports using results from the National Assessment of Educational Progress (NAEP) to compare the proficiency standards states set for their students. The latest report highlights the results of mapping state proficiency standards onto the NAEP scales using state assessment results from the 2018–2019 school year and the 2019 NAEP assessments for public schools.

Act 930 of 2017 Section 6-15-2907� (g) The State of Arkansas shall participate in the administration of the National Assessment of Educational Progress examinations.

Arkansas Division of Elementary and Secondary Education (DESE) Rules Governing the Arkansas Educational Support and Accountability Act (AESAA) of 1-1-2020

5.15 Selected public schools shall participate in any and all components of the National Assessment of Educational Progress (NAEP).
5.15.1 Any public school that fails to participate in the administration of any NAEP assessment shall be reported to the State Board and may be found in violation of the Standards for Accreditation of Arkansas Public Schools and School Districts and subject to action by the State Board.

Federal law specifies that NAEP is voluntary for every student, school, school district, and state. However, federal law also requires all states that receive Title I funds to participate in NAEP reading and mathematics assessments at fourth and eighth grades. Similarly, school districts that receive Title I funds and are selected for the NAEP sample are also required to participate in NAEP reading and mathematics assessments at fourth and eighth grades. https://nces.ed.gov/nationsreportcard/participating/

19 of 69

NAEP 2019 Reading

20 of 69

Grade 8 Reading Results

20

* The value is significantly different (p<0.05) from the value for the same jurisdiction in 2019.

Proficient

Basic

21 of 69

Grade 8 Reading Results

21

* The value is significantly different (p<0.05) from the value for the same jurisdiction in 2019.

Proficient

Basic

22 of 69

Grade 8 Reading Results

22

* The value is significantly different (p<0.05) from the value for the same jurisdiction in 2019.

Proficient

Basic

23 of 69

Grade 8 Reading Results:�2009-2019 : Subscale Effect Size Change

23

24 of 69

Grade 8 Reading Results

24

Black students 29 than White students
Hispanic students 10 than White students
Female students 13 than male students
Students who were eligible for free/reduced-price school lunch 24 than students who were not eligible
Arkansas’ average score higher than 4 states: DC, AK, AL, NM and not statistically significant from 13

25 of 69

NAEP Resources

26 of 69

Exploring NDE

26

NDE: https://www.nationsreportcard.gov/ndecore/xplore/nde
National and state data for all NAEP subjects

Civics, Economics, Geography, Math, Music, Reading, Science, Technology and Engineering Literacy, U.S. History, Visual Arts, Vocabulary, Writing

27 of 69

Item Map

Item Map analysis help readers understand student performance.
Example questions are “mapped” onto the NAEP scale for that subject

More difficult questions at the top of the map
Easier questions at the lower part of the map.

Complete scale range 0-500 (math, reading).

27

28 of 69

Item Map- Mathematics, grade 4

AR, 2019 Gender

Male (left)
Female (right)

IM link

29 of 69

NAEP Questions Tool

29

https://nces.ed.gov/NationsReportCard/nqt/Search

30 of 69

NAEP Questions Tool- Search

30

HSTI 2019 presentation: Innovative Digitally-Based Assessments

31 of 69

National and International Exams

Trends in International and Science Study (TIMSS)

First administered in 1995 (every 4 years); Grades 4 and 8
First digital assessment in 2019
TIMSS Released Questions

Program for International Studies (PISA)

First administered in 2000 (every 3 years); Age 15
Reading, mathematics, science and financial literacy
PISA Released Questions

The International Computer and Information Literacy Study (ICILS)

First administered in 2013 (every 5 years); Grade 8
Information literacy, critical thinking, technical skills, communication
ICILS Released Questions

The Progress in International Reading and Literacy Study (PIRLS)

First administered in 2001; Grade 4
Reading
PIRLS Released Questions

31

��

32 of 69

Dr. Laura Goadrich, ��Arkansas NAEP State Coordinator� laura.Goadrich@ade.arkanas.gov

https://tinyurl.com/mssht597

33 of 69

TIMSS Sample Questions

33

34 of 69

NAEP High School Transcript Study 2019

35 of 69

About the NAEP High School Transcript Study (HSTS)

Nationally representative sample of 2019 high school graduates

47,300 graduates
1,400 public and private schools

Data analyzed in report from 2019, 2009, 2000, and 1990
Links coursetaking patterns and NAEP grade 12 mathematics �and science results
Arkansas tested 13 high schools for HSTS 2019

35

36 of 69

HSTS Student Sample

Who’s included in HSTS analysis?

Graduates earning regular or honors diplomas
Graduated during school year of the NAEP assessment
Have 3 years of coursework on transcript
Earned at least 16 total credits, and positive English credit

36

37 of 69

37

38 of 69

Selected results from 2019

39 of 69

Compared to 2009 …

39

The number of credits earned by high school graduates in 2019 increased

Grade Point Average (GPA)

Grade point averages earned by high school graduates in 2019 increased

Curriculum Levels

The percentage of high school graduates who attained at least a midlevel curriculum increased in 2019

Average overall high school graduates NAEP Grade 12 mathematics assessment scores declined in 2019

There was no significant change in the average overall NAEP twelfth-grade science assessment scores in 2019

Coursetaking

NAEP Achievement

40 of 69

Credits

41 of 69

Course Types

Academic �Courses

English
Mathematics
Science
Social Studies
Visual and Performing Arts
World Language

Career and Technical Courses (CTE)

Includes courses in various subject areas such as:

Information technology
Human services
Business and Marketing
Communication and Technology

Other �Courses

Physical and health education
Religious education
Military science
Other miscellaneous courses

41

42 of 69

Highest total number of credits earned in 2019: 28

42

23.6*

26.2*

27.2*

28.1

* Significantly different (𝘱 < .05) from 2019.

43 of 69

Why are credits going up?

44 of 69

State Graduation Requirements in Mathematics and Science

44

Source: Data from 1987, 2000, and 2008 from CCSSO Key State Education Policies on PK-12 Education.�Data from 2019 from the Education Commission of the States- 50-State Comparison: High School Graduation Requirements.

Graduation requirements across states: 1987, 2000, 2008, 2019

States

45 of 69

Curriculum Levels

45

46 of 69

Curriculum Level Categories

46

Curriculum Level	Credits
Curriculum Level	English	Social Studies	Mathematics	Science	World Languages
Standard	4	3	3	3	0
Midlevel	4	3	3 Must include geometry �and algebra	3 Must include at least two of biology, chemistry, and/or physics	1
Rigorous	4	3	4 Must include precalculus �or higher	3 Must include biology, chemistry, and physics	3

47 of 69

Graduates are completing higher curricula in 2019

47

12

21

51

16

13

17*

47*

23*

9*

15*

37*

39*

4*

11*

26*

59*

0

20

40

60

80

100

Selected High School Graduation Year

2009

2000

1990

Percent of High School Graduates

Below Standard Standard Midlevel Rigorous

2019

* Significantly different (𝘱 < .05) from 2019.

48 of 69

Is there a specific course – or courses – that keep graduates from moving up to the most rigorous curriculum level?

49 of 69

Multiple requirements keeping graduates who attained a midlevel curriculum from moving into rigorous curriculum

49

* Significantly different (𝘱 < .05) from 2019.

50 of 69

Grade Point Average

51 of 69

Highest grade point averages earned for all course types in 2019

51

* Significantly different (𝘱 < .05) from 2019.

52 of 69

Highest grade point averages for selected racial/ethnic groups in 2019

52

* Significantly different (𝘱 < .05) from 2019.

53 of 69

NAEP Scores

54 of 69

Score declines in NAEP mathematics for graduates who attained a midlevel or rigorous curriculum �

54

Below Standard

Rigorous

Standard

Midlevel

156*

151

Overall

NAEP Advanced | 216

NAEP Proficient | 176

NAEP Basic | 141

300

* Significantly different (𝘱 < .05) from 2019.

55 of 69

Curriculum rigor is rising, GPAs are up, yet NAEP mathematics scores are down. Why is that?

56 of 69

Precalculus, Statistics, or Trigonometry as Highest-Level Course, by Race/Ethnicity: NAEP Mathematics Scores

56

* Significantly different (𝘱 < .05) from 2019.

‡ Reporting standards not met

‡

2009 2019 2009 2019 2009 2019 2009 2019 2009 2019 2009 2019 2009 2019

White

Black

Hispanic

Asian/

Pacific Islander

American Indian/

Alaska Native

Other

Overall

57 of 69

This study also collects information on students’ career and technical education courses. Any interesting findings there?

58 of 69

Percentage of Graduates in 2019 Who Earned At Least One CTE Credit

58

Career/Technical Course Subject	2019
One credit in any career/technical course	85
Information technology	29
Human services	28
Business and marketing	21
Communication and audio/video technology	17
Engineering and technology	14
Health care sciences	12
Agriculture, food, and natural resources	11
Hospitality and tourism	8
Architecture and construction	6
Public, protective, and government service	5
Manufacturing	4
Transportation, distribution, and logistics	4

* Significantly different (𝘱 < .05) from 2019

59 of 69

Percentage of Graduates in 2019 Who Earned At Least One CTE Credit

59

Career/Technical Course Subject	2019	2019 compared �to 2009	2009
One credit in any career/technical course	85		85
Information technology	29		38*
Human services	28		26*
Business and marketing	21		28*
Communication and audio/video technology	17		17
Engineering and technology	14		11*
Health care sciences	12		7*
Agriculture, food, and natural resources	11		8*
Hospitality and tourism	8		5*
Architecture and construction	6		8*
Public, protective, and government service	5		4
Manufacturing	4		5
Transportation, distribution, and logistics	4		5

Increase

Decrease

No significant change

* Significantly different (𝘱 < .05) from 2019

60 of 69

NCES Data Collections Span the Pandemic

60

61 of 69

NAEP School Surveys: �Qualitative Questionnaire Data from NAEP 2019 8^th grade mathematics

61

Circle size due to population of state (based on Census Bureau)

https://elsiejlee.com/

Sources: Pre-pandemic data is from the National Assessment of Educational Progress (NAEP) Mathematics Assessment. Pandemic access to internet data is from the 2021 Census bureau Household Pulse survey. Population data is from the Census Bureau. Pandemic population data is from 2020. Policy data is form Pew Charitable Trust and the National Governor's Association.

The 2019 data in the packed bubble chart is from NAEP 8th grade Mathematics. Students that participated in NAEP answered a question about if they had access to internet at home. The options they could choose were either a "Yes" or a "No". The 2021 data is from the Census Bureau Household Pulse data, which surveyed adults. If the adult had a child that was in public or private school, they answered a question about the availability of internet for educational purposes in their household. The responses to this question were on a scale from "Always Available", "Usually available", "Sometimes Available", "Rarely Available", or "Never Available." In order to match the NAEP data to the Census Bureau data, I aggregated "Always", "Usually", "Sometimes", and "Rarely" into one "Yes" category and used "Never" as the "No" category.

62 of 69

NAEP School Surveys: NAEP School Pulse 2021

62

Circle size due to population of state (based on Census Bureau)

https://elsiejlee.com/

Sources: Pre-pandemic data is from the National Assessment of Educational Progress (NAEP) Mathematics Assessment. Pandemic access to internet data is from the 2021 Census bureau Household Pulse survey. Population data is from the Census Bureau. Pandemic population data is from 2020. Policy data is form Pew Charitable Trust and the National Governor's Association.

The 2019 data in the packed bubble chart is from NAEP 8th grade Mathematics. Students that participated in NAEP answered a question about if they had access to internet at home. The options they could choose were either a "Yes" or a "No". The 2021 data is from the Census Bureau Household Pulse data, which surveyed adults. If the adult had a child that was in public or private school, they answered a question about the availability of internet for educational purposes in their household. The responses to this question were on a scale from "Always Available", "Usually available", "Sometimes Available", "Rarely Available", or "Never Available." In order to match the NAEP data to the Census Bureau data, I aggregated "Always", "Usually", "Sometimes", and "Rarely" into one "Yes" category and used "Never" as the "No" category.

63 of 69

Curriculum Level Categories

63

Curriculum Level	Credits
Curriculum Level	English	Social Studies	Mathematics	Science	World Languages
Standard	4	3	3	3	0
Midlevel	4	3	3 Must include geometry �and algebra	3 Must include at least two of biology, chemistry, and/or physics	1
Rigorous	4	3	4 Must include precalculus �or higher	3 Must include biology, chemistry, and physics	3

64 of 69

How have curriculum levels changed among different racial and ethnic groups?

65 of 69

Graduates are Completing Higher Curricula in 2019

65

* Significantly different (𝘱 < .05) from 2019.

Percentage distribution of high school graduates across attained curriculum levels: various years, 1990–2019

66 of 69

Precalculus, Statistics, or Trigonometry as Highest-Level Course, by Race/Ethnicity: Percent Completed

66

* Significantly different (𝘱 < .05) from 2019.

‡ Reporting standards not met (n<62).

67 of 69

Graduates Across Racial/Ethnic Groups Completing Higher Curricula in 2019

Below �Standard

Standard

Midlevel

Rigorous

67

* Significantly different (𝘱 < .05) from 2019.

68 of 69

No score change for NAEP Proficient graduates compared to 2009; declines for graduates below NAEP Proficient

Scores for:

Top mathematics GPA quartile steady
Lower mathematics GPA quartiles declined
Graduates completing calculus unchanged
Precalculus, statistics, and trigonometry, and below declined

68

	Reached or exceeded NAEP Proficient		Did not reach NAEP Proficient
	Group	Score change compared to 2009	Group	Score change compared to 2009
Mathematics GPA quartile	Top		Middle top
			Middle bottom
			Bottom
Highest mathematics course	Calculus		Precalculus, statistics, and trigonometry
			Algebra II
			Geometry
			Algebra I or below
9^th-grade mathematics course	Algebra II		Geometry
			Algebra I
			Below Algebra I

69 of 69

NAEP Mathematics Scores by Highest Math Class Completed

69

* Significantly different (𝘱 < .05) from 2019.

202

4.00 GPA in Calculus

Precalculus, statistics, and trigonometry

Algebra II

Geometry

Algebra I or below

NAEP Proficient

176 – 215

NAEP Advanced

216 – 300

Below �NAEP Basic

0 – 140

NAEP Basic

141 – 175

200

192

190

156

165

136

142

119

127

112

115

Calculus

2009 2019

YEAR

Highest Level Mathematics Courses