1 of 13

�

1

An Empirically-Based Systematic Design for Science Assessment

Linda Morell

University of California, Berkeley

STE2022 - 3rd International Conference on Science and Technology Education

October 6-7, 2022

FEUP, Porto - Portugal

"This material is based upon work supported by the National Science Foundation under Grant No. 2010322. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation."

2 of 13

Outline

2

Background

Construct Modeling Approach

Progress Variable
Items (questions)
Student responses
Measurement model

Summary

Next Steps

3 of 13

Systematic Design - Construct Modeling Approach

3

Progress Variable

Items

Measurement Model

Responses to Items

Construct map: An explanation of a theory of development or learning trajectory.

Items Design: The questions, performances, etc. asked of students that provide empirical evidence regarding the construct map

Outcome Space: A way of interpreting the responses and assigning them to a location on the construct map.

Measurement model: A way to empirically calibrate the construct map, items, scoring guides and students to create systematically designed assessment material.

The following exit criteria were used to make the decision to exit the inner-loop and continue with large-scale piloting using the measurement model:

1) Construct coverage: A sufficient number of items were developed so as to elicit responses representative of all levels of the construct map.

2) Exhaustive responses: Outcome spaces were revised until they reflected all distinct student responses generated from the piloting process.

3) Construct alignment: The construct and outcome space were revised so that the levels in both aligned to each other.

4 of 13

4

Science and Engineering Practices

Disciplinary Core Ideas

Crosscutting Concepts

5 of 13

NGSS: NEXT GENERATION SCIENCE STANDARDS

Disciplinary Core Ideas:

- Life Science

- Physical Science

- Earth & Space Science

- Engineering, Technology and the Application of Science

Crosscutting Concepts:

Patterns
Cause and effect
Scale, proportion, and quantity
Systems & system models
Energy & matter
Structure & function
Stability & change

Practices:

Asking questions & defining problems
Developing and using models
Planning & carrying out investigations
Analyzing & interpreting data
Using math & computational thinking
Constructing explanations & designing solutions
Engaging in argument from evidence
Obtaining, evaluating, and communicating information

6 of 13

Sources

6

1,237 middle school students piloted items
46 students participated in think-aloud Interviews

8-10 middle school teachers in Arizona and California
2 school administrators
12 members of the study team - content specialist, science educators, researchers

7 of 13

Construct Map of Student Understanding of Patterns in Ecosystems

7

Level of Performance	Description of Level
Level 3: Describes two or more changes completely	Finds and describes relevant patterns for more than one entity (i.e., population); describes how two or more patterns are related to each other; finds and describes subtle patterns (e.g., no change); recognizes trends
Level 2: Describes on change completely	Finds and describes patterns for one entity in an ecosystem; describes relevant patterns when prompted; describes discrete changes but has difficulty identifying subtle patterns.
Level 1: Describes one changes vaguely or incompletely	Describes graphs or maps in terms of symbols (e.g. dominant shapes or colors) only; provides vague descriptions of change.
Level 0: Naive conceptions	Does not identify a pattern; misinterprets the task; does not address the prompt; missing response or “I don’t know.”

8 of 13

Items Design: Sample Question

8

S12. What changes do you notice across the years in the figure above?

9 of 13

Construct Map, Item, Student Responses

9

Bands of understanding	Sample Student Responses to…What changes do you notice in the figure above?
Describes two or more changes completely	I noticed that in the 0 year, there was nothing after the fire. In the later years, it seems that the dirt layer has thickened. After the shrub stage, there were now trees growing there. In the final 2 stages, the roots of the trees has expanded. [complete description about changes in dirt and roots]
Describes one change completely	The ecosystem goes from nothing to small plants to large plants. [abstraction] The fire burns everything down to a bare field. It takes 1 or two years for grass. Another three or four years for shrubs and up to 150 years to finally get some real pine forest in the area.
Describes one change incompletely or vaguely	Plants are destroyed but new ones grow. After some years after the fire, trees started growing again and turned into a forest again. [describes only trees but not other smaller plants]
On task but provides an incorrect description of the patterns Off task or does not provide a meaningful response	it gets better and better [too vague] Over the course of several years, we can see how the forest is thriving after the wildfire. First, it starts out really small, but gradually we can see how small plants become big trees. [misinterpretation that smaller plants grow into big trees, rather than succeeded by] Science I don’t know / IDK

10 of 13

Measurement Model: Sample output

10

11 of 13

Measurement Model: Sample Output and Teacher’s Interpretation

11

“Yeah, he just surprised me… how he performed.

I mean, just looking at the data, it just gave me a lot of insights. Like, wow, this is interesting here.”

12 of 13

Application of the approach

12

Approach has been used to develop:

Data Collection Tools
Observation tools
Affective surveys
Academic tests

For a variety of different age groups:

Infant/toddler and preschool (3 months – 5 years old)
Elementary (5 – 10 years old)
Middle school (10– 13 years old)
High school (13-18 years old)
University and other adults

13 of 13

Thanks to the following collaborators:

Mark Wilson, PI, University of California, Berkeley

Kristin Gunckel, Co-PI, University of Arizona

Karen Tran, SERP Institute

David Dudley, SERP Institute

Anna MacPherson, American Natural History Museum

Sara Dozier, California State University, Long Beach

Sean Tan, University of California, Berkeley

Dan Moreno, University of Arizona

Malissa Hubbard, University of Arizona

Teachers, school administrators, and students who participated in this early part of the study

13

Thank you!

For more information, please email Linda Morell at lindamorell@berkeley.edu