1 of 33

School Environment and Leadership Research: Methodological Challenges and their Impact for Schools�

Prof. Rob Coe, Evidence Based Education

RPPL Brown Bag Conversation, November 2022

greatteaching.com

2 of 33

Overview

What are Evidence Based Education and the Great Teaching Toolkit?
Why does school environment and leadership matter for teachers’ practice and professional growth?
How trustworthy and helpful is the existing evidence on this?
How have we tried to make sense of this evidence in a way that is useful for school leaders?
What are the implications for organisations supporting schools and teachers?

2

3 of 33

Overview

What are Evidence Based Education and the Great Teaching Toolkit?
Why does school environment and leadership matter for teachers’ practice and professional growth?
How trustworthy and helpful is the existing evidence on this?
How have we tried to make sense of this evidence in a way that is useful for school leaders?
What are the implications for organisations supporting schools and teachers?

3

4 of 33

About EBE

Mission: “to improve learner outcomes, worldwide and for good”
What we do: Professional development, tools and services (user-centred, high-value, accessible, focus on equity)
Aligned with best available evidence, constantly innovating and evaluating, building new evidence

4

https://evidencebased.education/our-values/

5 of 33

What is the Great Teaching Toolkit?

A set of tools to

Help teachers understand and focus on the things that matter most for promoting student learning
Create the conditions in which all teachers can steadily learn to be even more effective in their practice
Provide rich, actionable feedback to support reflection, self-assessment, coaching & teacher learning

5

6 of 33

Great Teaching Toolkit: �Evidence Review

https://evidencebased.education/great-teaching-toolkit-evidence-review/

7 of 33

Elements of Great Teaching�What is worth learning for teachers?

01 Understanding the content

1.1 Deep and fluent content knowledge

1.2 Curriculum knowledge: sequencing

1.3 Knowledge of tasks, assessments, multiple explanations

1.4 Knowledge of student thinking: misconceptions

02 Creating a supportive environment

2.1 Relationships with students, cultural sensitivity

2.2 Student-student relationships & climate

2.3 Promoting learner motivation

2.4 High expectations, challenge and trust

03 Maximising opportunity to learn

3.1 Managing time and resources to maximise productivity

3.2 Clear and consistent rules, expectations, consequences

3.3 Preventing & responding to disruption, showing awareness

04 Activating hard thinking

4.1 Structuring: matching tasks, scaffolding, signalling objectives

4.2 Explaining: presenting & connecting ideas, modelling examples

4.3 Questioning: promoting hard thinking, assessing

4.4 Interacting: giving, receiving and responding to feedback

4.5 Embedding: practising, reinforcing & spacing learning

4.6 Activating: building independence, supporting metacognition

8 of 33

Overview of courses offered in the GTT

Foundation course�(over 5 weeks, to establish common understanding and language)

Introduce the framework
Basic Science of Learning
How the GTT works

Courses for leaders �(over 3 terms, working with colleagues to plan implementation)

Assessment Lead Programme
Science of Learning Programme
Behaviour and Culture Programme
Curriculum Lead Programme (coming soon …)

Courses for classroom teachers �(over 8 weeks, after identifying a specific focus for improvement)

D2: Creating a supportive environment
D3: Maximising opportunity to learn
4.1: Structuring
4.2: Explaining
4.3: Questioning
4.4: Interacting
4.5: Embedding
4.6: Activating

8

9 of 33

Why do teachers need more feedback?

Feedback supports learning in general (Hattie & Timperley 2007)
Feedback enhances task performance more generally (Kluger & DeNisi 1996)
Feedback enhances the power of goal setting (Locke & Latham, 2002)
Naturally available classroom feedback is not always reliable (Poor Proxies, Coe 2013)
Feedback may help to clarify/define specific development goals
Feedback may support coaching interactions

9

In tasks as complex as teaching, there is no expertise without experience.

But experience without good feedback is just repetition.

Feedback turns experience into learning.

10 of 33

Instruments to define, clarify and operationalise the elements

Student surveys

General screening tool
Detailed focus spotlights

Classroom observation tools

Video tools for sharing, evaluating & discussing

Staff surveys

School climate: environment and leadership

Assessments of teacher knowledge

Subject knowledge and PCK
Situational and behavioural questions

10

All validated against high-quality assessments of student learning

… when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science. (Thompson, 1891)

11 of 33

GTT feedback is

For volunteers only
Confidential to the person it concerns
Low stakes: no consequences
Designed to

Give insights (“Holding up a mirror”)
Operationalise the elements (“Clarifying what good looks like”)
Evaluate changes (“Showing progress”)

Consistent with multidimensionality and situatedness
Based on best existing practice & measurement theory
Measures validated & refined based on empirical data

11

12 of 33

Questions about EBE & GTT

What do you think of our theory of action? How could it be strengthened?
What other organisations are trying to do something similar? Who should we connect with?

12

13 of 33

Overview

What are Evidence Based Education and the Great Teaching Toolkit?
Why does school environment and leadership matter for teachers’ practice and professional growth?
How trustworthy and helpful is the existing evidence on this?
How have we tried to make sense of this evidence in a way that is useful for school leaders?
What are the implications for organisations supporting schools and teachers?

13

14 of 33

Elements of Great Teaching�What is worth learning for teachers?

01 Understanding the content

1.1 Deep and fluent content knowledge

1.2 Curriculum knowledge: sequencing

1.3 Knowledge of tasks, assessments, multiple explanations

1.4 Knowledge of student thinking: misconceptions

02 Creating a supportive environment

2.1 Relationships with students, cultural sensitivity

2.2 Student-student relationships & climate

2.3 Promoting learner motivation

2.4 High expectations, challenge and trust

03 Maximising opportunity to learn

3.1 Managing time and resources to maximise productivity

3.2 Clear and consistent rules, expectations, consequences

3.3 Preventing & responding to disruption, showing awareness

04 Activating hard thinking

4.1 Structuring: matching tasks, scaffolding, signalling objectives

4.2 Explaining: presenting & connecting ideas, modelling examples

4.3 Questioning: promoting hard thinking, assessing

4.4 Interacting: giving, receiving and responding to feedback

4.5 Embedding: practising, reinforcing & spacing learning

4.6 Activating: building independence, supporting metacognition

15 of 33

Classroom focus is necessary, but not sufficient

Teachers using the GTT model, courses & feedback work collaboratively (Ronfeldt, 2017)
An individual teacher in their classroom is supported or constrained by their (school) context (Bryk et al, 2010)
Professional growth & improvement depends on professional environment (Kraft & Papay, 2014)
These supports, constraints and environmental factors depend on school leaders

15

16 of 33

Overview

What are Evidence Based Education and the Great Teaching Toolkit?
Why does school environment and leadership matter for teachers’ practice and professional growth?
How trustworthy and helpful is the existing evidence on this?
How have we tried to make sense of this evidence in a way that is useful for school leaders?
What are the implications for organisations supporting schools and teachers?

16

17 of 33

School Environment and Leadership: Evidence Review

A Model for School Environment and Leadership

What school factors influence students’ learning, and why?

Methodological challenges in school leadership research

Why is the existing research problematic?

Evidence that school leadership and environment matter

What do the best studies tell us?

Methodology underpinning the evidence review

Technical details of our methodology

17

https://evidencebased.education/school-environment-and-leadership-evidence-review/

18 of 33

Three big problems in the leadership research

Correlations interpreted as causal levers
Poorly operationalised constructs
Advice that is neither meaningful nor actionable

18

19 of 33

Correlation ≠ Causation

Teacher/principal individual/collective efficacy:

“beliefs in one's capabilities to organize and execute the courses of action required to produce given attainments” (Bandura, 1997)

Many studies find correlations between teacher efficacy beliefs and student attainment (after controlling for other things), and interpret them (uncritically) as evidence that high efficacy leads to better attainment.
But

High student attainment / progress / value-added could cause teachers to believe that they are effective
Correlations are small and inconsistent anyway (Kim and Seo, 2018)
Interventions that raise efficacy perceptions lead to no increase in attainment (Jacob et al. 2015)

19

20 of 33

Jingle-jangle �(same word, different meaning, different word, same meaning)

Types of leadership

Instructional
Transformational
Distributed

(These are all meaningless / contested / vague)

We need

Clear definitions/operationalisations
Common usage across researchers
Validity evidence

20

21 of 33

Advice for leaders

Work with the energy at hand rather than try to resist it. Tap into your emotions and use them to your advantage.

Make sure that you do not let your incredible creative talent go to waste. Make a plan as to how you can make this happen during the days ahead.

21

Sources:

www.horoscopes.com

Leithwood et al (2020): Seven strong claims about successful school leadership revisited

22 of 33

What can the research tell us?

There is no convincing evidence that any training or support programme for school leaders makes them better

We don’t know how to train effective school leaders

There is no convincing evidence about the qualities, skills/knowledge, or behaviours of school leaders we should select for or promote

We don’t know how to select effective school leaders
We can’t give evidence-based advice to school leaders about what to do

It seems likely that

A school’s professional environment does make some difference to student outcomes, over and above the impact of individual teachers and school context

22

23 of 33

Questions about the research base

Do you agree that we cannot currently make clear, useful, generalisable claims about how to train, select or support school leaders?
If not, what studies would you point to?
If yes, what are the implications for training, selecting and supporting school leaders?

23

24 of 33

Overview

What are Evidence Based Education and the Great Teaching Toolkit?
Why does school environment and leadership matter for teachers’ practice and professional growth?
How trustworthy and helpful is the existing evidence on this?
How have we tried to make sense of this evidence in a way that is useful for school leaders?
What are the implications for organisations supporting schools and teachers?

24

25 of 33

Model for School Environment and Leadership

Students’ academic learning happens in classrooms, and depends on

The knowledge, skills, understanding, beliefs and behaviours of classroom teachers
The interactions between teachers, students and curriculum

These interactions are supported / enabled / constrained by wider school factors, which can be split into

Factors that support teachers
Factors that support students
Factors that support the curriculum

The time available for learning is also important
How well the school is managed as an organisation is important for the wider context in which these supports operate

25

26 of 33

The Model for Great Teaching

Great teachers:

01 understand the content they are teaching and how it is learnt

02 create a supportive environment for learning

03 manage the classroom to maximise �opportunity to learn

04 present content, activities and interactions that activate their students’ thinking

26

27 of 33

Our model: The classroom

27

Understanding the content

Activating hard thinking

Creating a supportive environment

28 of 33

Learning Supports (outside the classroom - mostly)

Family and community support
Student fundamental needs
Student beliefs and dispositions

Teacher collaboration
Collective teacher expertise
Professional learning

Curriculum goals and demands
Resources and materials

28

29 of 33

Learning Supports (outside the classroom - mostly)

29

Classroom time allocated
Time outside the classroom
Attendance at school
Disruption to timetabled lessons

30 of 33

Management factors

Supportive working relationships

Trust, support, safety, openness

Improvement mindset

Drive/belief in better, personal accountability, innovation, constructive evaluation

Delivery

Prioritisation, removing barriers, diagnosing & addressing problems, implementation

Staffing

Recruiting, retaining high quality, staff allocation & training

30

31 of 33

31

School Environment and Leadership Evidence Review

https://evidencebased.education/school-environment-and-leadership-evidence-review/

Staff survey tool �available NOW! �(for GTT subscribers)

32 of 33

Overview

What are Evidence Based Education and the Great Teaching Toolkit?
Why does school environment and leadership matter for teachers’ practice and professional growth?
How trustworthy and helpful is the existing evidence on this?
How have we tried to make sense of this evidence in a way that is useful for school leaders?
What are the implications for organisations supporting schools and teachers?

32

33 of 33

rob@evidencebased.education�@ProfCoe

33