What conditions support a more equitable and coherent system of science education?

Findings from ACESSE

Bill Penuel, Kerri Wingert,

Robbin Riedy, and Melissa Campanella

1

These slides are available at: http://bit.ly/ACESSEfindings

TIFFANY: Welcome -

BP: Welcome, introduce the team, ask folks to mute and introduce themselves in the chat. We are recording this zoom call, and there will be opportunities to ask questions and engage in the chat by clicking the chat button below.

Robbin and Melissa will be fielding questions in the chat, too. We also thank Deb Morrison, Tiffany Clark, and Gina Tesoriero for their work on data collection and initial analysis.

In collaboration with...

2

We are a research-practice partnership funded by the National Science Foundation.

We want to give special shoutout to our collaborators on this call, especially co-PI Tiffany Neill and Phil Bell.

We worked together with many of you and many of your predecessors to answer the question (NEXT SLIDE)

What makes for an equitable, coherent state system of science education?

ACESSE stands for….

Advancing Coherence and Equity in Systems of Science Education

3

BILL

4

Professional Learning Resources to Support NGSS Implementation

  • Co-designed by educators & researchers
  • Tested & refined over time
  • Easily shareable—over social media, email, paper

STEMteachingtools.org (web)

@STEMteachtools (twitter)

pinterest.com/stemeducation (pinterest)

BILL:

Much of the ACESSE work has focused on the co-design of tools, which can be found on the STEM Teaching Tools website.

Organizing for Equity and Coherence

PROMOTE EQUITY

Expand access to opportunities to learn

Share inclusive instructional practices

CRAFT COHERENCE

Build a shared vision for science teaching

Bring key components into alignment

ORGANIZE TOGETHER

Build a distributed team to lead

Network to share strategies & tools for change

5

BILL: (Define each equity, coherence, and organizational change)

Our mission as a research-practice partnership was three-fold.

We organized together, but how did we craft coherence and promote equity?

6

BILL:

A key idea is that to promote coherence and equity, there can be leverage points in a system that if you change them, they can drive a lot of other changes.

What makes for an equitable or coherent state system?

7

PROMOTE EQUITY

Expand access to opportunities to learn

Share inclusive instructional practices

In ACESSE: Culture-based pedagogies (interest and identity)

In ACESSE: Systems change so students everywhere encounter culture-based pedagogies

KERRI: The goals of our projects.

We supposed that we could support equity and coherence by enacting a theory of action that looked like this. (NEXT SLIDE)

What makes for an equitable or coherent state system?

8

Formative Assessment

ACESSE’s power comes from being able to work with CSSS because you reach into systems across the country, and our goal was to develop tools and research that supports instructional change and culture-based teaching methods.

What makes for a coherent system?

Coherence - where all the parts are moving in a common direction

  • vertical - people and their ideas
  • horizontal - assessments, tools, materials

9

We wanted equity, and we want it across our system. We call this coherence; when all the parts are moving in the same direction. I love physical science, and I often think of this in terms of aligned and unaligned magnetic fields; the magnet as a whole is strongest when you get the fields lined up.

Research

What makes a system more equitable?

Vertically coherent?

Horizontally coherent?

Use the chat - what do you think?

10

KERRI: What are the things that you think matter the most in making a system equitable?

Our research goals are to understand the systems that cubers work in and to interpret what makes systems more equitable, vertically and horizontally coherent.

The literature suggests some “ingredients”:

  • Vision-Building Routines with Educators
  • Infrastructure at the state level
  • Collective Efficacy
  • Constancy : “Vision held by leadership over time.”

11

Kerri:

As I talk these through, please join a side conversation if you’d like about how you are doing these things in your state.

Vision-Building Routines

  • reach of PD
  • number of hours of aligned PD

12

stemteachingtools.org/pd

Reach = number of people

Hours

You can find these resources at stemteachingtools.org/pd

Infrastructure

  • a system of people that work to lead in an aligned way
  • includes standards and policies

(Cohen, Hopkins, Spillane, Woulfin)

13

Kerri:

Cutting-edge examples of infrastructure that we are seeing nationally: MINK & CO, #NGSSChat

Can also be official including regional science specialists who are paid to be innovators and conduits of innovation. How much of the NGSS did you adopt? Just the evidence statements? THe full foundation boxes? This can provide infrastructure for implementation.

Can also be policies.

Collective Efficacy

  • how much control you think you have

  • filter-down effect to teachers in school systems (Goddard & Goddard, 2001)

14

Low collective efficacy leads to “a solemn satisfaction with the status quo” (Donohue, Hattie, & Eels, 2018)

KERRI:

Comes out of Bandura’s work in the 70s on confidence; the more confident you were, the better you seemed to do, even if you and the less confident person had the same actual capacity. We see this manifested in the power pose literature; the ways that you envision your effectiveness actually seems to influence your effectiveness.

More recently, Goddard & GOddard did a study where they looked at teacher effectiveness and their leaders’ expressions of effectiveness, and even when they “controlled for” SES; teachers were more effective when they had leaders who gave a sense of collective efficacy.

There’s a great quote that I like a lot from Donohue, Hattie and Eels, 2018 *READ*. This has implications for equity since the status quo in the US is a White-cismale dominated STEM education fields.

We investigated collective efficacy by looking at how effective folks said their team was, and also in how case study respondents described problems.

Constancy

Quality over time

  • Not just the “7 year rule”
  • Keeping the vision in leadership over time

15

KW: we made up a category - emergent category that arose from analysis.

Cubers have expressed many times the rule, I think it comes from Bret Moulding, of 7 years.

16

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Collective Efficacy

Constancy

?

PAUSE HERE for a conversation in the chat - what questions do you have about these four elements?

What is present in your own system?

BP: Explain the model; this is not a typical regression; it uses QUALITATIVE data to analyze state systems.

KW: Along those lines, look at the dark green boxes on the left. Which of these do you feel your state has MORE of. Use the chat box to write… which of these do you think your state has in place and why?

How did we decide “in a set?”

We calibrated:

  • 77 interviews
  • 30+ survey questions
  • deciding “how in is it?”
  • infinite patience and generosity

17

KW: This analysis is comparative. THat means that we looked across the 7 states and said, “This group has 3D standards and this group doesn’t, and this subgroup has an assessment that is 3D too. They are more in the set”

Melissa can share what it means to be a 0.2 or a 0.4 or a 0.66.

18

State

Vision- Building Routines

Collective Efficacy

Constancy

Infrastructure

Vertical Coherence

Horizontal Coherence

Vision for Equity

J

0.5

1

0

0.5

0.4

0.2

0.5

U

0

1

1

0.5

0.8

0.8

1

S

1

0.5

1

0.5

0.6

0.6

0.5

H

1

0.5

1

1

0.8

0.2

0.5

R

0.5

0.5

1

1

0.6

0.8

0.5

N

0

0

0

0

0.4

0.4

0

C

0

0

0

0

0.6

0.6

0

MELISSA

19

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Collective Efficacy

Constancy

?

PAUSE HERE for a conversation in the chat - what questions do you have about these four elements?

What is present in your own system?

BP: Explain the model; this is not a typical regression; it uses QUALITATIVE data to analyze state systems.

KW: Along those lines, look at the dark green boxes on the left. Which of these do you feel your state has MORE of. Use the chat box to write… which of these do you think your state has in place and why?

What configurations support equity?

20

21

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

22

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Collective Efficacy

Constancy

KW:

23

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

24

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

What made for vertical coherence?

25

26

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

MELISSA

27

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

WK:

28

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

WK:

29

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

WK:

What made for horizontal coherence?

30

31

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

32

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

33

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

34

Equity

Vertical Coherence

Horizontal Coherence

Vision-Building Routines

State-Level Infrastructure

Constancy

Collective Efficacy

KW:

35

What is your reaction to these results?

What do you want to know more about?

36

We started with district/school literature; we’ve added to that in these ways by showing these components are salient for an SEA.

The implications are that we hope you go to act on this conceptual resources in ways you can enact leadership.

ACESSE as a project with resources sought to help you build these 3 outcomes through the 4 inputs.

OPEN QUESTION - How do you support collective efficacy building?

RR- map ACESSE resources to the inputs

Bottom line:

Work with what you got!

37

KW: There are a lot of concoctions, both accidental and the product of resilience.

Equity

That states that had pathways to equity incorporated equity work throughout everything.

Collective efficacy was the shortest “route” to equity.

38

RR

Horizontal coherence is an especially long game.

39

Photo by Cagatay Orhan on Unsplash

What pathways might work for you, given your strengths?

PAUSE HERE for a conversation in the chat - what questions do you have about these four elements?

What is present in your own system?

BP: Explain the model; this is not a typical regression; it uses QUALITATIVE data to analyze state systems.

KW: Along those lines, look at the dark green boxes on the left. Which of these do you feel your state has MORE of. Use the chat box to write… which of these do you think your state has in place and why?

Intentional strategy.

Good people to work with.

Frustration, resilience, humor, and pride.

Collective efficacy.

41

KW: I want to add one note that we saw across the interviews is that there’s an intense amount of depth in the ways Cubers interpreted your work and act, the interveiws reflected very deep intention. You also all expressed support and confidence in the teams you worked with. You also shared a lot of personal emotion, including frustration, resilience, humor and even some pride in your work.

Building collective efficacy might be a place to go next.

KW: collective efficacy isn’t one that we have talked about much. What does it mean to a sense of leadership give the obstacles that we can face? When you are the only person in your whole state with a similar job? When you are aware of “gaps” but the systemic barriers aren’t systemically addressed? So it may be worth, organizationally, thinking about what collective efficacy means in light of equity.

The ways that interviewees in this study talked about their own ability to have an impact -- has to do with how we understand our constraints and this includes political ones.

Photo by Matt Howard on Unsplash

ACESSE Resources to Support Equity and Coherence

42

Constancy

Collective Efficacy

Infrastructure

Routines

Tuning protocols

Vision Survey; Focus Groups

Network Mapping

PD Resources (A-F, STT); Practical Measures

Audience question--can you think of other ACESSE Tools/Resources you’ve used that have helped build toward equity and coherence?

Tuning Protocols- build vision, help teams work together on the same page, reduce tension and turnover

Vision Survey and Focus Groups- help teams get on the same page, learn what is happening in all levels of the system to address concerns, coordinate action

Network Mapping- find and connect with diverse stakeholders, building buy-in for policy, standards, and assessments

PD Resources- build PD routines around shared resources helps coherence

Practical Measures- routine measurement toward your goals

References

Goddard, R. D., & Goddard, Y. L. (2001). A multilevel analysis of the relationship between teacher and collective efficacy in urban schools. Teaching and Teacher Education, 17(7), 807–818. https://doi.org/10.1016/S0742-051X(01)00032-4

43

Acknowledgment

This material is based in part upon work supported by the National Science Foundation under Grant Number DRL-1561300. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

44

How does the math work?

1+1 = 1

TRUE and TRUE = TRUE

if(and((membergroup1 = TRUE, (membergroup2 = TRUE), TRUE, FALSE)))

45

KW: This analysis is comparative. THat means that we looked across the 7 states and said, “This group has 3D standards and this group doesn’t, and this subgroup has an assessment that is 3D too. They are more in the set”

What do we mean by cross-level routines alignment- of people and materials?

46

Teams review curricular materials for alignment

Effort is spent gathering assessment input (*TN: Is this about alignment?)

Teams offer a large amount (npeople*hours) of ACESSE goal-oriented PD in early years of project

Average per state for Question 29 case study survey "Ask teachers how well-aligned..." ORANGE ONES

State report - Bring key people together in our state who influence what the state assessment looks like AND ...what classroom assessment...

Collaborative Alignment Work -- to Develop Deep Understanding of Targets

BP: “Crafting Coherence”

TN: Building capacity

What do we mean by infrastructure?

47

  • 3D standards are in place (Per NSTA and interviews)
  • a 3D assessment is in place (per interviews and a study of national assessments)
  • leaders have
  • meetings are convened of regional leaders
  • meetings are rated as helpful in supporting systems change
  • PD providers are communicated with regularly

Collaborative Alignment Work -- to Develop Deep Understanding of Targets

BP: “Crafting Coherence”

TN: Building capacity

What do we mean by collective efficacy?

48

Teams feel that they have influence over state science system

Sources of incoherence are in the state leaders' sphere of influence

Teams feel that they work effectively as a state team

Questions 35 (Columns P-U in KW's excel sheet n = 37, rawdata tab); "earliest" measure of state efficacy

Sources of incoherence are in the state leaders' sphere of influence (T1 & T2 only)

May 2018 survey of team members N=37 survey questions about team work AL-AQ

This i s based on Elmore’s work

TN: Guess; “District leaders and classroom teachers would say the same things at different levels.”

Is third column… which state team are we talking about?

What do we mean by constancy?

49

There is low turnover in the leadership

Consistency of team attendance at ACESSE meetings

Beliefs about 3D from vision scale for 3D

Primary ACESSE member has influenced state science system for 5 years or more(as of May 2019)

primary state leader attended all 3 ACESSE 13 meetings?

Score from 2018 report from vision scale (in state report & 181203 excel sheet from KW)

TN: Make sure you include cases of people who have constancy and vision, but when leaders leave what happens?

Cases: PA, SD,

Relationship to other non-ACESSE states as a conclusion.

What do we mean by 3D state-level infrastructure?

50

State team function

Framework-aligned state standards

3D State Assessment in Place

PD providers are communicated with about 3d/ efforts made to educate wide array of PD providers

Meetings of similarly-positioned leaders

Formalized leader group who are accountable to/ aligned with state leader (accountable to state leader)

Q17 Please answer how often you have engaged in the following activities in the last twelve months.

NSTA website

Several sources

a few questions in the May 2018 survey

could combine with >>>

could combine with <<<

*TN: Make sure that 3D is emphasized; not just “state-level infrastructure” but this is about the vision

Texas - would it have state-level infrastructure?

Column rating: 1 - 0 - 0 - 1 -0 - 0 - 1

What predicts equity?

51

Consistency

Coverage

States

Configuration associated with equity

1. Constancy*Infrastructure

1

0.5

R, H

2. Collective Efficacy

1

0.33

J, U

3. Routines*Constancy

1

0.5

S, H

Total

1

0.83

52

53

BP Talks VERY short here; each pathway is covered in more detail in the next 3 slides.

0.83 solution coverage is good. It means that the three pathways explain 83% of the cases, or all but one of the states. 4 out of 7 states follow this, and there are two MORE states that DON’T follow this pathway and DON’T have the outcome of equity, so they are said to be included in the model here.

Consistency is “the extent to which a causal combination leads to an outcome” (Elliott, 2013). “If you have a super high consistency, but your coverage is super low, then your solution isn’t that compelling because it doesn’t describe many cases at all.” (Elliott, 2013, p.6) We have a low-ish consistency but we also have a low number of cases in general.

For fuzzy sets, it is calculated a bit strangely and doesn’t make sense if you are looking at just what is on the screen. You have to know what is in the table, because it uses the MINIMUM values within each column. It is calculated as the sum of the (MINIMUMS of the (membership score for causal combo) and Y is the MINIMUM of the (membership score for the outcome set)) taken and divided by the sum of the X.

Then we chose a cutoff of 0.8, which is acceptable (Ragin 2007).

High consistency means that combination almost always leads to the outcome, and it is 1.0 in all three configurations. These pathways were consistent in yielding equity. S & H BOTH had constancy and routines, and equitable progress. Thus it has a 1.0 consistency.

Coverage is for ONLY the rows that fit the solution, and it “represents how many cases with the outcome are represented by a causal condition” (Elliott, 2013, p.5). It is calculated using fuzzy-set MINIMUMS from the Truth Table, so the math is not straightforward. It is calculated by taking the sum of the [ (MINIMUMS of the X or (membership score for causal combo) and MINIMUM of the Y or (membership score for the outcome set)) ] taken and divided by the sum of the Y or membership score for the outcome set.

If you have constancy in leadership you get two pathways + routines OR +infrastructure

These are alternate pathways hwo states got here.

If you don’t have a lot of PD TIME then focus on infrastructure, then think about

If you’re new and have a lot of turnover you have to hang your hat on collective efficacy

54

BP: Talk this one

If you have constancy in leadership you get two pathways + routines OR +infrastructure

These are alternate pathways how states got here.

If you don’t have a lot of PD TIME then focus on infrastructure, then think about

If you’re new and have a lot of turnover you have to hang your hat on collective efficacy

What predicts vertical coherence?

55

  • Configuration 1 - Presence of constancy and infrastructure, absence of influence. 36% coverage. Consistency 100%
    • States R & H
  • Configuration 2 - Collective efficacy, absence of routines and infrastructure. 21% coverage, consistency 89.9%
    • States J & U
  • Configuration 3 - Presence of constancy and routines, absence of influence. 36% coverage, consistency 100%
    • States S & H

56

What predicts horizontal coherence?

Configuration 1 - presence of collective efficacy and constancy, absence of routines and infrastructure covered 13.89% of the (model)

Configuration 2 - presence of constancy and infrastructure and absence of influence covered 33.33%

Configuration 3 - presence of routines and constancy but absence of influence covered 33.33% of the model

57

*Constancy is in all configurations

This is a hopeful set of “recipes”

What don’t we have in this model? “What other way can you know? It’s a closed space for outsiders.”

“There’s no other way to get this data.”

58

Continuous effort - when you get something aligned then something changes

In constancy you are developing system knowledge because you know what can get in the way.

airig case - one person was there to oversee the process for a long time; it’s not surprising this is necessary because we don’t see a lot of coherence systems.

We need constancy in leadership and vision and personnel, so this is why horizontal coherence is rare.

What makes for an equitable or coherent state system? - Google Slides