Manaiakalani Inquiries
Using meta-analysis to identify what works
All inquiry cycles are intended to:
What do we need from you?
Student data
Gather evidence:
At this stage you will be gathering evidence about your students:
Self review for gathering evidence:
Hypothesise and Research
You will be forming hypotheses about likely barriers to student learning
You will be forming hypotheses about teaching actions likely to accelerate student learning
These hypotheses will be generated from the evidence about the students combined with evidence about the learning environment
The curly self review questions:
And then onto researching the hypotheses...
You will be using a range of research to consider the relationships between the students’ needs and the instruction they receive
You will be researching specific refinements to practice that are most likely to address particular issues given the profile of the students
You will be considering digital possibilities that address the issues
The research self review questions:
Leaders of inquiries: Supporting quality inquiries
Solutions that arise for strong analysis of the issue
Watch out for those where solutions generated before problem investigated
Evidence about student learning
Make sure that student learning can be assessed
A strong theory
Interrogate the logical links between the identified issue and the proposed refinements
Intensity of implementation
Make sure changes to the teaching can be documented and evidenced
Changes to learning process
Make sure that the inquiry has addressed what will change in the learning process for the students, and that this can be documented
22th March
COL teachers inquiry meeting
What is a “causal chain”?
A causal chain is when a cause leads to an effect and that effect becomes the cause of another effect
A leads to B. B leads to C. C leads to D.
Any intervention you design will (consciously or not) be based on a causal chain you have in mind - this is your theory of action.
The logic of what you are doing:
THE CHAIN OF EVENTS
Achievement challenge:
General mathematics
Outcome:
Language in mathematics
Specific language outcome:
Justifying and reasoning in mathematics
Intervention:
Teaching that impacts on the specific language outcome
How and why will the specific outcome impact the general achievement challenge?
How and why will the teaching impact the specific outcome?
Why is thinking about a causal chain important before you design your intervention?
Your theory needs to be as strong as possible before you test it. It needs to be reasonable, plausible, and consistent with what we already know - so that it is most likely to work.
If A leads to D - but doesn’t lead to B and C- that means that your intervention worked, but not for the reasons you theorised!
To really understand deeply how teaching (as a cause) leads to a valued student outcome (effect), we need to know about more than just the first cause in the chain and the final - we need to know about each link.
Soon, we will ask you to articulate your theory. That is to name the different links in your causal chain and explain why you think A will lead to B and why B would lead to C?
Talk time 1 Articulate your theory - make it obvious
1)If I do xxxx in my teaching, then students will do xxxx .
If students do xxxx, this will improve their maths reasoning.
Improved maths reasoning, will lead to better achievement in maths because.....
2) Teacher knowledge building: how does your articulated theory reflect the TKB that you have been doing?
How do your language measures reflect the TKB that you have been doing?
How do your new ways of teaching reflect the TKB that you have been doing?
Do you have the right measures that match your theory?
Designing measures that match your theory:
A general measure- Standardised measures across the COL (e.g. PAT mathematics scale score relative to norm). This is a “far” effect measure.
Particular measures- subtest analyses of a standardised measure? (e.g. a sub-score of an aspect that would be most sensitive to changes in language use)
Particular measure - language in use analysis? Developed by researcher? Doesn’t need to be a test - needs to be a baseline measure. This is a “near” effect measure because it is the first change we might expect to see after the intervention
Talk time two - what will your “near” measure need to tell you?
What will you need to know about each part to find out whether your intervention worked?
Achievement challenge:
General mathematics
Outcome:
Language in mathematics
Specific language outcome:
Justifying and reasoning in mathematics
Intervention:
Teaching that impacts on the specific language outcome
How and why will the specific outcome impact the general achievement challenge?
How and why will the teaching impact the specific outcome?
Inclusion and exclusion criteria
Meta-analyses use inclusion and exclusion criteria so that the studies that are aggregated are relevant and of a “good-enough-standard”.
We will need to know that you have systematically tested your chain of events, this requires information about each of the boxes (in the previous slide)
- What changed in your teaching?
- What changed for the students?
-What changed in the specific outcome?
- What changed in the more general measure of achievement?
3rd May/ 15th May
From planning to implementing
Outcomes for today:
Implementation
At the end of the inquiry cycle, you want to be able to attribute any changes in learner outcomes (in part) to changes in the experiences of learners
To do this you need to be able to demonstrate exactly how the learning experience has changed for learners
Some form of “repeated measure” of teaching over time
This could be a detailed description, with evidence, of what teaching was like prior to the intervention compared to what it was like during or after the intervention
There will likely be variable levels of “implementation fidelity” - that is, the actual intervention will likely not be exactly like the planned intervention. Low levels of implementation fidelity can be a good thing i.e. because you are inquiring and refining what you had planned.
What is important though is that you can clearly show a reader what you did differently so he/she can judge whether or not it is reasonable to think that changed learner outcomes are related to these changed practices
Implementation: Discussion points
What is the intended change in teaching? What will the teacher do differently? How much?
When? Over how long? With whom?
How will you know if the teacher has done something differently over time?
How will you know if the intended changes are the same as the actual changes?
How will you, as leader, support these understandings with your teachers?
Ideas for checking implementation:
Monitoring
As well as the kinds of informal monitoring you do lesson-by-lesson and minute-by-minute, it is useful to plan some more formal checkpoints
Such checkpoints allow you to check in a systematic way how learners are experiencing the intervention and whether it is beginning to have the impact on learner outcomes that you what it to have
Micro-formative assessments used at set intervals can be very useful here. The mini asTTle reading tests or ARBs are examples of published “small” assessments you could use to gather information about changes in specific aspects. Or you could design your own approaches for making judgments about progressions (e.g. LLP or ELLP). Or you could collect and analyse writing samples of a few “case study” students at set intervals.
Student voice is also vital (after all, we are making changes on their behalf!). Collecting this in regular and manageable ways could involve quick-fire “exit” questions for all learners in a class, or more extended chats with a few case study teachers
Monitoring?
What is the intended change for a learner?
How are they experiencing the intervention
How will they engage differently? How will you know?
How will you know whether the intended ‘treatment’ (sorry about the language) was the same as the actual
How will their outcomes begin to shift? How will you know?
LEADERS: How will you support your teachers to monitor changes for learners?
Ideas for easy monitoring?
In your groups
COL teachers 31 May
Monitoring evidence sharing
and interrogation
Discussion 1:
Evidence about the monitoring YOUR implementation
What methods have you used to collect information?
Discussion 2:
Evidence about the monitoring of STUDENT learning
Discussion 3:
Evidence about the changes to the intervention
July 26
Term 3
The accelerator term
Planning and predicting: THEORY is everything.
Horn, B. & Freeland Fisher, J. (2016) A blueprint for breaking through: Federally funded education research in 2016. Report for the Clayton Christensen Institute for Disruptive Innovation.
OK - Theory is NOT QUITE everything.
Education may not be an exact science, but it is too important to allow it to be determined by unfounded opinion, whether of politicians, teachers, researchers or anyone else.
The only worthwhile kind of evidence about whether something works in a particular situation comes from trying it out. Arguments from theory are simply no match for something that has been tried and tested.
Coe, (1999), Manifesto for Evidence-Based Education (http://www.cem.org/attachments/ebe/manifesto-for-ebe.pdf)
Integrity in implementation
What we need is less fidelity of implementation (do exactly what they say to do) and more integrity of implementation (do what matters most and works best while accommodating local needs and circumstances).
Conversation 1:
Recap - what research evidence are you using to back up what you are doing?
Recap - how is that ‘research thing’ operationalised (brought to life) in your intervention?
Has your learning design been true to the theoretical intent?
3. What have you learned so far?
Thinking about your integrity of implementation:
Analysing our Implementation data
Moving from:
To:
To:
Conversation 2 - Implementation
What was the intended change in teaching? What did the teacher do differently? How much?
When? Over how long? With whom?
How do you know if you have done something differently over time?
Were intended changes in your teaching the same as the actual changes?
Boiling it down...opportunities for teacher learning
1 - Figure out the students’ strengths and needs
(did you get this right?)
2 - Use the existing research base to plan something different that is likely to use strengths to meet the need
(did you get this right?)
3 -DO the different thing
(did you do this right?)
4-Which engages the students in a different way of learning
(did you get this right?)
5-Which results in learning.
(for all? Some? Why??? What explains the anomalies?)
So - analysing how does it work?
Have you have tested out your theory of what would happen?
Is it working?
OR
Is it partly working?
-Working for some?
-Working a little bit?
-Working a little bit and just for some?
The logic of what you are doing:
THE CHAIN OF EVENTS
Achievement challenge:
General mathematics
Outcome:
Language in mathematics
Specific language outcome:
Justifying and reasoning in mathematics
Intervention:
Teaching that impacts on the specific language outcome
How and why will the specific outcome impact the general achievement challenge?
How and why will the teaching impact the specific outcome?
The causal link....
Student can do
Students’ learning processes over time
Teachers do
Teachers’ learning over time
How does it work? - Getting inside the ‘mechanisms’ - finding the ANOMALIES
Analysing how it’s working
Is it partly working?
-Working for some?
-Working a little bit?
-Working a little bit and just for some?
WHY?? What theory explains this?
THIS WILL TELL YOU WHERE IN YOUR CAUSAL CHAIN TO REFINE.....
Horn, B. & Freeland Fisher, J. (2016) A blueprint for breaking through: Federally funded education research in 2016. Report for the Clayton Christensen Institute for Disruptive Innovation.
So - how to accelerate? Challengers Qs
Does the implementation have integrity (teacher did it well? For whom, under what conditions?)
Does the intervention have integrity? For whom? Under what circumstances?
Conversation 3
The causal link.... Accelerate
Given what you know about:
Challenge this term
Use your data to learn something:
The causal link....
Student can do
Students’ learning processes over time
Teachers do
Teachers’ learning over time
Leaders day: Term 3
The leadership role in inquiry?
Support and challenge teachers in processes of inquiry
Expect commitment to teacher learning
Draw connections between inquiries for whole school/ team learning
Build collective improvements to practice
Horn, B. & Freeland Fisher, J. (2016) A blueprint for breaking through: Federally funded education research in 2016. Report for the Clayton Christensen Institute for Disruptive Innovation.
Integrity in implementation
What we need is less fidelity of implementation (do exactly what they say to do) and more integrity of implementation (do what matters most and works best while accommodating local needs and circumstances).
Teachers’ Conversation 1:
Recap - what research evidence are you using to back up what you are doing?
Recap - how is that ‘research thing’ operationalised (brought to life) in your intervention?
Has your learning design been true to the theoretical intent?
3. What have you learned so far?
Thinking about your integrity of implementation:
Conversation 1
Leadership overlay:
Recap - what research evidence are you using to back up what you are doing?
Recap - how is that ‘research thing’ operationalised (brought to life) in your intervention?
Has your learning design been true to the theoretical intent?
Thinking about your teachers’ integrity to theory:
Analysing our Implementation data
Moving from:
To:
To:
Conversation 2 - Implementation
What was the intended change in teaching? What did the teacher do differently? How much?
When? Over how long? With whom?
How do you know if you have done something differently over time?
Were intended changes in your teaching the same as the actual changes?
Leadership overlay - Implementation
What was the intended change in teaching?
What did the teacher do differently?
How much?
When?
Over how long?
With whom?
How do you know if you have done something differently over time?
Were intended changes in your teaching the same as the actual changes?
Leadership challenge and support:
Boiling it down...opportunities for teacher learning
1 - Figure out the students’ strengths and needs
(did you get this right?)
2 - Use the existing research base to plan something different that is likely to use strengths to meet the need
(did you get this right?)
3 -DO the different thing
(did you do this right?)
4-Which engages the students in a different way of learning
(did you get this right?)
5-Which results in learning.
(for all? Some? Why??? What explains the anomalies?)
The causal link....
Student can do
Students’ learning processes over time
Teachers do
Teachers’ learning over time
How does it work? - Getting inside the ‘mechanisms’ - finding the ANOMALIES
Analysing how it’s working
Is it partly working?
-Working for some?
-Working a little bit?
-Working a little bit and just for some?
WHY?? What theory explains this?
THIS WILL TELL YOU WHERE IN YOUR CAUSAL CHAIN TO REFINE.....
Horn, B. & Freeland Fisher, J. (2016) A blueprint for breaking through: Federally funded education research in 2016. Report for the Clayton Christensen Institute for Disruptive Innovation.
So - how to accelerate? Challengers Qs
Does the implementation have integrity (teacher did it well? For whom, under what conditions?)
Does the intervention have integrity? For whom? Under what circumstances?
Conversation 3
The causal link.... Accelerate
Given what you know about:
Leadership overlay 3
The causal link.... Accelerate
Given what you know about:
Leadership support and challenge?
Challenge this term
Use your data to learn something:
The causal link....
Student can do
Students’ learning processes over time
Teachers do
Teachers’ learning over time
Summary - Leadership implications?
Discussions with inquirers - knowledge building and strengthening
How will you know as a leader, whether your actions have worked as intended?
How will you know what new knowledge teachers have gained?
Structures for conversations?
Individual discussion? Or collaborative? How to support critique at your place?
What approaches could you use to disseminate learnings from multiple inquiries?