Using AI to Promote Equitable Science Teaching and Learning

​

Attending to Diversity, Equity, Inclusion, and Justice in Our Work and Relationships�

Kevin C. Haudek

Opportunity to learn

No Child Left Behind mandated improving science education by providing learning opportunities for all students, regardless of gender, ethnicity, socioeconomic status, disability, and English-language proficiency.

Individual learners develop understanding in different ways and every student requires equal opportunities and resources to develop deep knowledge of science.

Framework for K-12 Science Education

Focus on integration of three dimensions of science learning

• Disciplinary Core Ideas (DCIs)
• Scientific and Engineering Practices (SEPs)
• Crosscutting concepts (CCCs)

Challenge: hard to assess using multiple choice (MC) format (Krajcik, 2021) but using open response formats leads to evaluation challenges

Framework for K-12 Science Education

Focus on learning development over time

• Aligning curriculum, instruction, and assessment along empirically validated learning pathways

Learning Progressions (LP) describe complex thinking focused on student progression towards a deep understanding of a topic as reflected in the ability to apply knowledge when explaining phenomena and solving real-life problems (NRC, 2012).

Developmental Approach

Challenge: implementing requires the capability to provide quick and meaningful feedback on student performance with respect to LP levels to all stakeholders (students, teachers, administrators)

Opportunity requires immediate scoring & feedback

In order to support all students:

• They need to engage in authentic & complex scientific practices
• They need formative feedback during learning & engagement
• They need support in timely fashion from teachers

Challenge for teachers due to time, number of students, training in 3D science

​

AI-based scoring of student work can help address these challenges!

Overview of AI based scoring using supervised learning

Student responses

Expert codes

Feature extraction

ML model training

ML Model

New student responses

Predicted codes

Predicted codes valid?

No. Iterate

Yes

Compare

Potential issues in AI-enabled scoring of science assessments

Issues we need to address with AI scores:

• Validity of scores
• Bias
• Range of student responses & scores
• Interpretability of scores

Guiding Questions for this session

1. How does AI-based automatic and timely feedback promote opportunities to learn for all students?

​

• How do we ensure that AI-based scoring and feedback is not biased against one group?

�

These projects focus on using AI-based scoring of formative, 3D science tasks to support teaching and learning in the classroom.

Session overview

1. Supporting Instructional Decision Making: The Potential of An Automatically Scored Three-dimensional Assessment System

--Presenters: Xiaoming Zhai and Namsoo Shin

1. Evaluating Effects of Automatic Feedback Aligned to a Learning Progression to Promote Knowledge-In-Use

--Presenter: Leonora Kaldaras

1. Discussant: Okhee Lee
2. General Question & Answer

​

This material is based upon work supported by the National Science Foundation (Grants 2200757, 2101104, 2100964, 2101166 & 2101112). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the supporting agencies

Employing Automatic Analysis Tools Aligned To Learning Progressions To Support Opportunity To Learn In NGSS Classroom

CADRE PI Meeting 2023

Leonora Kaldaras, Kevin Haudek, Joe Krajcik

How can Learning Progressions Support Opportunity To Learn ?

​

LPs are grounded in Developmental Approach

• Learning requires alignment of curriculum, instruction and assessment (Duschl et al., 2007)
• What is developmentally appropriate depends on prior opportunities to learn
• LPs help diagnose student level of understanding and guide educators to support transition to the higher levels
• help adjust instruction to the needs of individual students by helping identify specific learning opportunities needed

How can Learning Progressions Support Opportunity To Learn ?

• By helping adjust instruction to the needs of individual learners
• Only possible given quick and accurate LP-aligned feedback to students and teachers is provided
• AI can help by providing quick
• LP level placement
• LP-aligned feedback
• Important to ensure AI-based scores are valid with respect to LP

Evaluating Effects of Automatic Feedback Aligned to a Learning Progression to Promote Knowledge-In-Use

Goal: study the effect of automatic feedback on LP-aligned formative assessments on student learning outcomes in the context of NGSS curriculum

Main features:

• Context: NGSS-aligned curriculum for 9th grade Physical Science
• Previously validated LP
• Immediate feedback provided to individual students on models and explanations at multiple points during the year
• Class summaries provided to teachers

Award # 2200757

Kevin Haudek (PI), Michigan State University

Joseph Krajcik (co-PI), Michigan State University

Leonora Kaldaras (Co-PI), Stanford University & University of Colorado Boulder

Interactions Curriculum

https://interactions.open3d.science/home

–Aligned to NGSS

–For 9^th Grade Physical Science

–Driven by Phenomena

• Experience Phenomena
• Raise Questions
• Investigate by engaging in 3D learning
• Develop models and explanations & raise questions

​

Increasing complexity

Previously validated LP for

​

Electrical Interactions combining the Coulomb’s law and Energy ideas

1. Kaldaras, L., Akaeze, H., & Krajcik, J. (2021). Developing and validating Next Generation Science Standards‐aligned learning progression to track three‐dimensional learning of electrical interactions in high school physical science. Journal of Research in Science Teaching, 58(4), 589-618.
2. Kaldaras, L., Akaeze, H., & Krajcik, J. (2021). A methodology for determining and validating latent factor dimensionality of complex multi-factor science constructs measuring knowledge-in-use. Educational Assessment, 26(4), 241-263.
3. Kaldaras, L. (2020). Developing and Validating NGSS-Aligned 3D Learning Progression for Electrical Interactions in the Context of 9th Grade Physical Science Curriculum. Michigan State University.

Automatic Scoring of NGSS LP-aligned Formative CR Assessments

Challenges for ML scoring include developing rubrics that can:

• accurately capture the complex 3D nature of student understanding of LP-aligned items
• yield high inter-rater reliability (IRR) between human and machine scores
• allow to design highly specific LP-aligned feedback
• yield valid scores beyond simple human-machine agreement

​

In LP validation: multi-level, holistic rubrics (Haudek et al., 2012; Kaldaras et al., 2021)

• Easier to present 3D nature of the item

​

In automatic scoring: analytic rubrics (Liu et al., 2014; Moharreri et al., 2014; Sieke et al., 2019)

• series of binary rubrics that identify the presence or absence of relevant ideas
• Yield slightly better human-machine agreement (Jescovitch et al., 2020; Wang et al., 2021)
• Easier to evaluate score validity with respect to LP (Kaldaras & Haudek, 2022)

​

Currently, there is no research available on how to deconstruct an LP- aligned holistic rubric for scoring NGSS LP-aligned CR assessments into an analytic rubric suitable for ML scoring approaches while preserving the 3D nature of the rubric

​

The picture shows two wood cars on a frictional surface with metal sheets attached. Both metal sheets are negatively charged. The wedges prevent the cars from moving. When the wedges are removed, the carts will move. Predict which direction they will move and when they will stop. Use ideas about forces and energy as appropriate.

Level 0 (0 points): Energy and/or Coulomb’s law are mentioned but not used meaningfully

​

Level 1 (1 point): explain why the carts will repel but not when they will stop and why

Level 2 (2 points): explain why the carts will repel and when they will stop and why using EITHER Coulomb's law OR Energy ideas. Answers might contain inaccuracies

​

Level 3 (3 points): explain why the carts will repel, when the carts will stop and why by using BOTH ideas related to Coulomb’s law and Energy

Holistic Rubric

DCI: Types of Interactions (Coulomb’s law, HS-PS2-4)

SEP: Constructing Explanations

CCC: Cause and Effect

Overview of Analytic Rubric Categories

Category 1: state which direction the carts will move

Category 2: explain why the carts will move in the stated direction

Category 3: recognize that the carts will stop

Category 4: explain when the carts will stop using Coulomb's law

Category 5: explain when the carts will stop using Energy

Category 6: integration of Coulomb’s law and Energy to explain when the carts will stop and why

Category 7: inaccuracies in the student response or the lack of evidence in the response for student ability to integrate dimensions of NGSS

Kaldaras, L., Yoshida, N., & Haudek, K. Rubric Development For AI-Enabled Scoring Of Three-Dimensional Constructed-Response Assessment Aligned to NGSS Learning Progression. In Frontiers in Education (p. 927). Frontiers.

• Identify the smallest possible aspects of 3D knowledge that can be meaningfully described for a given category
• Ensure that all the important aspects of the phenomenon in question are reflected in the analytic rubric

Sample Student Responses and Associated Analytic Scoring

Category 1: state which direction the carts will move

Category 2: explain why the carts will move in the stated direction

Category 3: recognize that the carts will stop

Category 7: inaccuracies in the student response or the lack of evidence in the response for student ability to integrate dimensions of NGSS

Possible feedback: in your response relate ideas of electric force and energy to distance and explain when the carts will stop and why

​

Sample Student Responses and Associated Analytic Scoring

Category 1: state which direction the carts will move

Category 2: explain why the carts will move in the stated direction

Category 3: recognize that the carts will stop

Category 4: explain when the carts will stop using Coulomb's law

Possible Feedback: Consider adding ideas related to energy to your explanation of when the carts will stop

Sample Student Responses and Associated Analytic Scoring

Category 1: state which direction the carts will move

Category 2: explain why the carts will move in the stated direction

Category 3: recognize that the carts will stop

Category 6: integration of Coulomb’s law and Energy to explain when the carts will stop and why

Possible Feedback: I like how you related ideas of energy to electric force and tracked energy conversion to explain when the carts will stop.

Methods and Results

Analytic Scoring: 2 scorers, Cohen’s Kappa for each category >0.8 (Cohen, 1960)

Data: 1253 responses from 9th grade Physical Science

ML Model:

• Supervised ML
• Constructed Response Classifier tool (CRC; Jescovitch et al., 2020) based on on RTextTools (Jurka et al., 2012)
• Compare the human and machine assigned scores, report the model performance:

​

Kaldaras, L., Yoshida, N., & Haudek, K. Rubric Development For AI-Enabled Scoring Of Three-Dimensional Constructed-Response Assessment Aligned to NGSS Learning Progression. In Frontiers in Education (p. 927). Frontiers.

Conclusion

NGSS-aligned LPs can support opportunity to learn

• adjust instruction
• Help students reflect on their responses

Only possible given quick and accurate LP-aligned feedback

AI can help by providing quick:

• LP level placement
• LP-aligned feedback

​

​