1

Thursday, May 12 | 1–2:30 p.m. EDT

College and Career-Ready Standards:

Are teachers implementing them in the classroom

and are they improving student learning?

Welcome & Logistics

Morgan Polikoff

Researcher, C-SAIL

Associate Professor, USC Rossier School of Education

Logistics

• Audio muted. We have muted attendee audio for the duration of this opening session.

​

• Questions. Please use the chat to submit questions.

​

3

Overview of C-SAIL

Andy Porter

Director, C-SAIL

Professor emeritus

University of Pennsylvania GSE

​

About C-SAIL

Center on Standards, Alignment, Instruction, & Learning

• Funded by Institute for Education Sciences in July 2015
• How are college- and career-ready (CCR) standards implemented, do they improve student learning, and what instructional tools measure and support their implementation?

​

Partners

• University of Pennsylvania, Graduate School of Education
• American Institutes for Research
• University of Southern California
• Vanderbilt University
• University of Delaware
• Four partner states: California, Massachusetts, Ohio and Texas; Philadelphia

Scope of research

• Longitudinal study of the effects on student achievement using NAEP

​

• Implementation study of federal and state academic content standards-based policies

​

• New conceptions and measurement strategies of academic content

​

• Building and testing an intervention to move academic content standards-based reform beyond the classroom door

​

​

Context

• At the beginning of our work standards-based reform had been the primary policy approach to improving K-12 education for three decades.

​

• Standards-based reform at the federal level began with No Child Left Behind with its heavy emphasis on accountability.

​

• In 2015 the federal Every Student Succeeds Act had just been approved with its focus on College and Career Ready Standards and greater policy discretion for states.

​

Distinctive C-SAIL design elements

• Guided by the policy attributes theory
• Attributes: specificity, consistency, authority, power, stability

​

• Seeks to clarify any differences in the quality of implementation and effects for students with disabilities, English language learners, all other students

​

• Variety of methodologies including large-scale standardized student achievement testing, large-scale surveys, randomized control trials, case studies.

The Effects of States’ Adoption of CCR Standards on Student Achievement

Mengli Song

Co-director, C-SAIL

Principal Researcher, American Institutes for Research

Study Context, Purpose, & Design

• Context: All 50 states and DC adopted new CCR standards in math and ELA between 2007 and 2015.

​

• Purpose: assess the effects of states’ adoption of CCR standards on student achievement as measured by NAEP.

​

• Design: Comparative interrupted time series design
• Approach: Compare change in student achievement trend from before to after the adoption of CCR standards between treatment (T) states and comparison (C) states, where:

• T states = states with less rigorous prior standards

• C states = states with more rigorous prior standards

• Data: state-level NAEP data in reading and math from 1990 to 2017
• Sample: N = 26 ~ 34 states that adopted CCR standards in 2010

​

10

Effects of CCR Standards: Grade 4 Reading

NAEP Scores for Treatment States  

​

11

ES: 1-year effect = -0.06* (p < .05); 3-year effect = -0.08* (p < .05);

5-year effect = -0.10^† (p < .10); 7-year effect = -0.10^† (p < .10)

Effects of CCR Standards: Grade 8 Reading

12

NAEP Scores for Treatment States

​

Effects of CCR Standards: Grade 4 Math

NAEP Scores for Treatment States

13

ES: 1-year effect = 0.01; 3-year effect = -0.03; 5-year effect = -0.06; 7-year effect = -0.08

Effects of CCR Standards: Grade 8 Math

NAEP Scores for Treatment States

14

ES: 1-year effect = 0.01; 3-year effect = -0.03; 5-year effect = -0.07; 7-year effect = -0.10* (p< .05)

Summary

States’ adoption of CCR standards did not lead to improved student achievement in reading or math during the time period examined.

15

Implementation Study�& Findings

Amy Stornaiuolo

Researcher, C-SAIL

Associate Professor, Reading/Writing/Literacy

Graduate School of Education

University of Pennsylvania

How were CCR Standards Implemented?

• Surveys
• 2015 & 2020; teachers, principals, district leaders
• Interviews
• 2015-2020; state and district leaders
• “Deep Dive” district case studies
• 2019-2020; interviews, focus groups, observations

17

Partners: California, Kentucky, Massachusetts, Ohio, Texas, Philadelphia (PA)

Overview

• Considerable variation in how standards were implemented—across states, districts, schools, and classrooms

​

• Despite widespread challenges to implementation, teachers generally think the content in the standards is good for children

​

• Evidence of concrete actions that states, districts, and schools can take to improve implementation:

​

• Improve the specificity of guidance.
• Develop consistent state and district infrastructure.
• Provide aligned professional learning for teachers.

​

18

#1: Improve Specificity of Guidance

What are the top five resources teachers indicated they wanted to support their instruction? (2020)

There were no changes from 2015 in the top 5 resources that teachers wanted.

Educators want more specificity in guidance, but states and districts are reluctant to provide too much specificity because of concerns about local control.

#1: Improve Specificity of Guidance

• How districts provided more specificity was key
• High specificity + low flexibility = educators felt lack of autonomy
• Low specificity + high flexibility = educators felt unanchored

​

• Flexible specificity was powerful for strong educator buy-in and motivated enactment (which are tied to implementation quality and student outcomes)
• Clear guidance & supports
• Opportunities for local feedback (e.g., data-driven inquiry cycles)
• Clear policies about adaptations

#2: Develop consistent state and district infrastructure

• Specific and aligned curricula
• Effective PD
• Stable standards and assessments
• Clear, transparent policies and communication about policy changes
• Adaptive leadership models
• “Soft” power - less punitive, more supportive

21

Variation in alignment is tied to the kinds of infrastructure provided by states and districts and its consistency.

#3: Provide aligned professional learning

• PL: sees teachers as professionals involved in decision-making processes
• Individualistic and collaborative opportunities (e.g., coaching, PLCs, pathways)
• PL as a mechanism for building consistency, authority and specificity of standards through classroom, school, district level opportunities

22

Professional learning opportunities for educators should be aligned with curriculum materials, responsive to teachers’ needs and expertise, and collaborative

Summary

• On the ground implementation of standards-based instruction continues to be a challenge, with broad variability in instructional quality

​

• Several actions districts can take to move the needle:

​

• Provide flexibly specific guidance
• Develop stable infrastructure
• Offer aligned professional learning opportunities

23

CCR Standards and Students with Disabilities

Douglas Fuchs & Lynn S. Fuchs

C-SAIL Researchers

Vanderbilt University

and American Institutes for Research

Three Major Conclusions: Career- and College-Ready Standards for Students with Disabilities

• Longitudinal Study
• NAEP analysis on effects of CCR standards on reading and math achievement

​

• Implementation Study
• Interviews with SEA officials
• Surveys with district administrators, principals, and teachers

25

Conclusions

• NAEP analysis provides little basis for thinking challenging academic standards boost the reading or math performance of SWD.

​

• Interview and survey data suggest lack of effects may be exacerbated by lack of buy-in on the idea of universal applicability of standards and need for more knowledge about appropriate instructional practices to support SWD and their teachers.

​

• Interview data reveal some encouraging initiatives at the state and district levels to support coordination between GE and SE to address needs of SWD in standards reform.

​

​

​

​

​

​

​

26

Should We Re-Envision Standards Reform to Include CCRS Instructional Reform?

• Differentiating classroom instruction to address the needs of students at different instructional levels in ways that forge meaningful instructional engagement in challenging content for a broader range of learners.

​

• Supplementing classroom instruction with structured small-group intervention for at-risk learners such as SWD, which simultaneously addresses grade-level challenging standards while attending to foundational skills.

​

• Provide teachers the means to realize the promise of challenging academic standards.

27

CCR Standards and English Learners (ELs)

Nelson Flores

Researcher, C-SAIL

Associate Professor, Educational Linguistics Division, Penn Graduate School of Education, University of Pennsylvania

Increasing Centralization of EL Policy

• States have taken the lead in shaping the EL accountability policy infrastructure including in developing or choosing assessments that will be used to measure English language proficiency

​

• Most states have partnered with one of two national consortia (WIDA and ELPA21) and use their assessment system for identifying, monitoring, and reclassifying Els

​

• This increasing centralization has been welcomed rather than seen as a threat to local control because of its connection with authority as opposed to power

29

The Increasing Rigor of EL Classification

• With increasing alignment of English language proficiency assessments with general state assessments more potential ELs have been classified as actual ELs and must meet more challenging criteria to be reclassified

​

• “False positives”: placed unnecessarily in EL program
• “Reentry program”: prematurely removed from EL program
• “Long Term English Learners”: in EL program throughout their entire schooling career’

​

• Some states (including PA) are taking advantage of the new flexibility offered by ESSA to move toward a more fluid classification and reclassification process

30

Teachers Perspectives on ELs across Partner States

2020—CA, OH, TX

• OH teachers report feeling significantly less prepared to teach ELs than teachers in CA or TX (Ohio has not improved)
• CA and TX teachers reported receiving significantly more PD than OH for ELs
• EL teachers are providing comparable grade-level instruction to mainstream teachers in CA, OH and TX

31

A Typology of EL Policies across Districts

Contextual factors such as density, demographics, and history played a significant role in the capacity of states and districts to support ELs

32

Creating Instruments to �Measure Alignment

Morgan Polikoff

Researcher, C-SAIL

Associate Professor, USC Rossier School of Education

Need for new measures of instruction

• Intervention study teachers needed to be able to report on their instruction and compare with their state states’ CCR standards.

​

• Data needed to be fine-grained enough as to provide useful feedback to teachers in the intervention study.

​

• Instrument needed to be feasible enough for teachers to complete on a regular basis.

​

34

Surveys of Enacted Curriculum

35

Ensuring adequate content coverage

• Expert review and revision of existing content languages

​

• Recommended revisions:
• Adding specific topics
• Editing existing topics
• Revising cognitive demand levels
• Adding separate reporting for SWDs & Els

​

• Cognitive interviews
• Topic and cognitive demand revisions worked well
• Separate reporting for SWDs and ELs did not

​

36

Trying the instrument

• 79 teachers, half math/ELA
• Every other week log
• End-of semester survey
• 1 week of assignments/assessments

​

Results

• Moderate alignment of logs with surveys
• Underreporting cognitive demand coverage on surveys
• Underreporting coverage of high-frequency topics on surveys
• Reliability of content analyses of assignments was high in mathematics, lower in ELA

​

37

Small-scale pilot

• 11 teachers, half math/ELA
• 6 months
• Logs every few weeks, feedback as planned for intervention study

​

Lessons learned

• Weighting of “big” vs. “small” content standards
• Multiple raters/adjudicating disagreement
• Revisions to log process

​

​

38

Feedback on Alignment and Support for Teachers (FAST) Study

Toni Smith

Researcher, C-SAIL

Principal Researcher, American Institutes for Research

Goal of the FAST Program

40

Provide teachers with support to:

• Understand the standards
• Implement aligned instruction to support all students in reaching the standards

Grade 4: Math

Grade 5: English Language Arts

Components of FAST Program

41

Personalized Coaching

​

-Collaborative team

meetings

-Individual sessions

Artifacts for Feedback

​

-Video recordings of

own instruction

-Instructional logs

Online Library of Resources

​

-Documents that unpack the standards

-Lesson activities

Study design

42

FAST Implementation

• Although FAST was made available to all teachers in the target grades, not all participated
• 35 teachers in the instructional impact sample for math (64%)
• 24 teachers in the instructional impact sample for ELA (69%)

​

• Over two years, teachers participated in ~6 team meetings and 5 coach sessions, and they completed ~5 or 6 logs
• Participation was a bit higher in math than ELA.
• Teachers participated in about half the total number of sessions initially planned

​

• Participants reported finding the FAST activities helpful for reflecting on and improving instruction

​

​

​

43

Year 2 Impact

44

What may explain the results?

• Lack of association between alignment and achievement
• Instructional alignment as measured by our SEC-based survey was not associated with student value-added scores in our data.

​

• Alignment as a necessary, but not sufficient condition for improving achievement
• In addition to alignment, teachers may need to improve other aspects of instruction (e.g., questioning) to see impacts on achievement.

​

• Need for curricular support
• Teachers may need support in using curriculum to design aligned lessons.

​

45

Discussion

Dan Goldhaber

C-SAIL advisor

Director of CALDER

Q&A

Extra slides after here

48

C-SAIL Overview�& Guiding Framework

• Andrew C. Porter
• Director and Principal Investigator, C-SAIL
• Professor Emeritus, Penn Graduate School of Education, University of Pennsylvania

Our guiding framework is the policy attributes theory.

It posits that there are five features that make a policy successful:

1. Specificity
2. Consistency
3. Authority (law, expertise, consistency with social norms, charismatic leader)
4. Power
5. Stability

​

More details on page X of handout.

​

How did we assess the effects of CCRS?

• Approach: Comparative interrupted time series (CITS) design
• Focused on student outcome trend before and after the adoption of CCRS.
• Did the change in trend differ between treatment (T) and control (C) states?
• Classification of T and C States: Based on the rigor of each state’s pre-CCRS standards as measured by:
• Prior Rigor Index (0-7): (Carmichael et al., 2010)
• Prior CCSS-Similarity Index (1-5): (Schmidt & Houang, 2012) 
• T states = states with less rigorous prior standards
• C states = states with more rigorous prior standards
• States with certain index values were excluded to create a sharper T-C contrast.

​

51

Analysis Samples and Model

• Samples for Achievement Analyses
• Restricted to states that adopted CCRS in 2010
• For analyses based on Prior Rigor Index, N = 29 for reading and 34 for math
• For analyses based on CCSS-Similarity Index, N = 26 (math only)
• Sample for Graduation Rate Analysis
• Restricted to states that adopted CCRS in 2010 or 2011
• N = 38, based on Prior Rigor Index
• Analytic Model
• “CITS baseline trend model”
• Controls for state-level NAEP exclusion rate, per pupil expenditure, % of students eligible for free- or reduced-price lunch, % of minority students, and pupil-to-teacher ratio.

​

52

NAEP Trends Before and After the Adoption of CCRS in Treatment and Comparison States

53

2010, year of CCRS adoption

T states: C states:

Trends in High School Graduation Rates in Treatment and Comparison States

54

Why weren’t results positive?

• Potential Reasons
• CCRS may be no more effective in improving student achievement than prior standards.

​

• CCRS may not have been well implemented, with slow rollout (typically 3-5 years) and a multitude of implementation challenges.

​

• Study results need to be interpreted with caution due to design limitations (e.g., lack of a true no-treatment comparison group, selection bias not fully accounted for, and less-than-perfect alignment between NAEP and CCRS).

​

55

C-SAIL Implementation Team

56

Meghan Comstock

Adam Edgerton

Nelson Flores

Erica Saldívar García

Shira Korn

Amy Stornaiuolo

Morgan Polikoff

Katie Pak

Phil Nichols

Mark Lewis

Victoria Glock-Molloy

The Policy Attributes Theory

57

What was our research design and data sources?

• Surveys
• Interviews
• “Deep Dive” case studies
• Partner states:
• California, Kentucky, Massachusetts, Ohio, Texas
• Philadelphia, PA

58

How do the policy attributes differ across respondents and how did they change over time?

59

There was moderately high buy-in for standards across the board among teachers, principals, and superintendents, but buy-in was significantly higher for principals and/or superintendents. Some respondents reported significantly higher authority, consistency and power in Year 4.

How did teachers vary in their authority�(buy-in) to the standards?

60

There was moderately high buy-in for standards across teachers of math, ELA and ELs in elementary and high school, but buy-in was significantly lower for teachers of SWDs.

• There were no changes in patterns of teacher buy-in from Year 1 to Year 4.

What are the top five resources teachers indicated they wanted to support their instruction? (2020)

61

There were no changes over time in the top 5 resources that teachers wanted.

According to teachers, what are the top five challenges to their instruction? (2020)

62

There were no changes over time in the top 5 challenges to teachers’ instruction.

Do teachers align their instruction to the content emphasized in the standards? �Has this alignment increased over time?

63

There is wide variation in teachers’ alignment to the standards by content area and grade level.

What predicts aligned instruction?

• Specific and aligned curricula
• Effective PD
• Stable standards and assessments

64

Variation in alignment variation is tied to the kinds of infrastructure provided by states and districts.

How does specificity work to support implementation of the standards?

“Of course the state pushes about college and career readiness, but I think it’s more to the local entities to really standardize that, articulate it, and create thoughtful plans towards that. It’s like the state provides the overall umbrella, but then it's up to the districts… to really, implement more defined plans.” (TX District Leader)

65

Though specificity is desired on the ground, states and districts are reluctant to provide too much specificity because of concerns about local control.

How is power (rewards and sanctions) playing out under ESSA?

State officials stated that accountability changes under ESSA have allowed them to adopt a “smart power” message that is less punitive and more supportive

66

How are districts supporting teacher implementation of standards?

• Individualistic and collaborative professional learning
• Adaptive leadership practices
• PL as a mechanism for building consistency, authority and specificity

67

PL and adaptive leadership play a key role in building authority, specificity and consistent in standards implementation.

EL Implications

• Simply covering grade-level standards may not be sufficient at increasing EL academic achievement

​

• Creating a state-level system of differentiated support for districts based on the typology proposed may be a way of supporting districts in raising EL academic achievement

​

• Creating a research agenda to identify other contextual information that should be considered in conceptualizing this differentiated support can help create a more robust system of differentiation

​

​

68

Goal of the FAST Program

69

Provide teachers with support to:

• Understand the standards
• Implement aligned instruction to support all students in reaching the standards

Grade 4: Math

Grade 5: English Language Arts

Components of FAST Program

70

Personalized Coaching

​

-Collaborative team

meetings

-Individual sessions

Tools to Support Reflection

​

-Video recordings of

own instruction

-Instructional logs

Online Library of Resources

​

-Documents that unpack the standards

-Lesson activities

Research Questions

1. How is the FAST program implemented?

​

• Does the FAST program lead to greater alignment of teachers’ instruction with state standards?

​

• Does the FAST program lead to stronger student achievement?

​

71

Study Design

72

5 Districts

3 States

In each district, randomly assign schools to:

Treatment

(29 Schools)

Control

(27 schools)

56 Schools

Timeline and Data Collection

73

Study and Impact Samples

• We focused the analysis of the impact on instruction on teachers present in the spring of year 2
• For math, the instructional impact sample included 47 of 55 schools (85%) and 108 of 142 teachers (75%)
• For ELA, the instructional impact sample included 43 of 52 schools (87%) and 73 of 108 teachers (68%)

​

• For achievement, we focused on students present in the spring of year 2.
• For both math and ELA, almost all students had achievement data

​

​

​

​

​

​

74

#4 Power matters

State officials stated that accountability changes under ESSA have allowed them to adopt a “smart power” message that is less punitive and more supportive

75