Designing exam questions that target scientific practices
Crystal Uminski
cuminski2@unl.edu
We will be using �Poll Everywhere. Please sign in using the QR code �or visit pollev.com/crystalu239
Learning Goals
3
Behaviors scientists engage in to understand the world
Foundational ideas within a discipline
Interdisciplinary �ways of thinking�
NRC (2012); NGSS Lead States (2013)
4
Scientific Practices
Planning Investigations
Asking Questions
Analyzing Data
Using Models
Using Mathematics
Engaging in Argument
Evaluating Information
5
n =309 exam items
Uminski & Couch (under review)
6
Uminski & Couch (under review)
7
There are 140,000 species of moths and butterflies and 7,000 species of caddisflies. Propose a hypothesis about the impact of herbivory on adaptive radiations in insects and explain how this hypothesis could be tested.
modified from Urry et al. (2017)
Uminski & Couch (under review)
8
3. What compounds make up the “backbone” of DNA?
6. The diploid number of chromosomes in a human skin cell is
Uminski & Couch (under review)
Breakout room: �
What are challenges or barriers that make testing scientific practices difficult in your class?
How can we address challenges of assessing practices?
10
Use data and phenomena that have already been curated
Use the 3D-LAP as guidelines for writing new items
Use the 3D-LAP to identify where existing items can be improved
Use closed-ended items to streamline grading
11
https://datanuggets.org/
Free lesson guides �and curated data sets
https://learn.concord.org/
12
http://datanuggets.org/search-current-data-nuggets/
13
14
https://concord.org/ngss/
15
What are some physical aspects that engineers who build coasters may have to take into consideration when designing coasters?
16
How can we address challenges of assessing practices?
Use data and phenomena that have already been curated
Use the 3D-LAP as guidelines for writing new items
Use the 3D-LAP to identify where existing items can be improved
Use closed-ended items to streamline grading
18
3D-LAP includes criteria for:
Scientific practices in constructed response items
Scientific practices in selected response items
Crosscutting concepts
Disciplinary core ideas
19
Developing and Using Models
Student is given or asked to construct a mathematical, graphical, computational, symbolic, or pictorial representation and use it to explain or predict an event, observation, or phenomenon.
Scientific Practices Criteria for Constructed Response Assessment Tasks
20
Developing and Using Models
Student is given or asked to select a mathematical, graphical, computational, symbolic, or pictorial representation and select an appropriate explanation or prediction about an event, observation, or phenomenon based on the representation.
Scientific Practices Criteria for Selected Response Assessment Tasks
Use the 3D-LAP to help you write questions�
21
Using Mathematics and Computational Thinking
Student is asked to use mathematical reasoning or a calculation and interpret the results within the context of the given event, observation, or phenomenon.
Identify phenomenon for your question(s)�
22
Brown anoles are a species of lizard. Smaller anoles (≤ 25 mm body length) are not as fast as larger anoles (>25 mm body length). Researchers predicted predators might apply selective pressure on the anoles because they can more easily prey upon the smaller lizards.
To test this prediction, the researchers did a mark-recapture study in July and October of the same year. The data they collected is in the table.
Use the 3D-LAP to help you write questions�
23
Using Mathematics and Computational Thinking
Student is asked to use mathematical reasoning or a calculation and interpret the results within the context of the given event, observation, or phenomenon.
Ask students to do a calculation�
24
Brown anoles are a species of lizard. Smaller anoles (≤ 25 mm body length) are not as fast as larger anoles (>25 mm body length). Researchers predicted predators might apply selective pressure on the anoles because they can more easily prey upon the smaller lizards.
To test this prediction, the researchers did a mark-recapture study in July and October of the same year. The data they collected is in the table.
Use the 3D-LAP to help you write questions�
25
Using Mathematics and Computational Thinking
Student is asked to use mathematical reasoning or a calculation and interpret the results within the context of the given event, observation, or phenomenon.
Ask students to interpret the data�
26
Brown anoles are a species of lizard. Smaller anoles (≤ 25 mm body length) are not as fast as larger anoles (>25 mm body length). Researchers predicted predators might apply selective pressure on the anoles because they can more easily prey upon the smaller lizards.
To test this prediction, the researchers did a mark-recapture study in July and October of the same year. The data they collected is in the table.
Use the 3D-LAP to help you write questions�
27
Using Mathematics and Computational Thinking
Student is asked to use mathematical reasoning or a calculation and interpret the results within the context of the given event, observation, or phenomenon.
Ask students to connect the data to the “big picture”�
28
Brown anoles are a species of lizard. Smaller anoles (≤ 25 mm body length) are not as fast as larger anoles (>25 mm body length). Researchers predicted predators might apply selective pressure on the anoles because they can more easily prey upon the smaller lizards.
To test this prediction, the researchers did a mark-recapture study in July and October of the same year. The data they collected is in the table.
29
How can we address challenges of assessing practices?
Use data and phenomena that have already been curated
Use the 3D-LAP as guidelines for writing new items
Use the 3D-LAP to identify where existing items can be improved
Use closed-ended items to streamline grading
30
Developing and Using Models
Student is given or asked to select a mathematical, graphical, computational, symbolic, or pictorial representation and select an appropriate explanation or prediction about an event, observation, or phenomenon based on the representation.
According to the evolutionary model above, �which is the oldest common ancestor?
Does this item elicit evidence of a �scientific practice?
31
Developing and Using Models
Student is given or asked to select a mathematical, graphical, computational, symbolic, or pictorial representation and select an appropriate explanation or prediction about an event, observation, or phenomenon based on the representation.
✓
x
According to the evolutionary model above, �which is the oldest common ancestor?
x
✓
32
Does this item elicit evidence of a �scientific practice?
Suppose that you have 10 grams of carbon and 4 grams of hydrogen available. Do you have enough starting materials to produce 14 grams of methane (CH4)? If yes, explain why and how much methane you can make. If no, explain why not and show the maximum amount of methane that could be prepared. Be sure to include a balanced chemical equation and all calculations needed to support your answer to receive full credit.
Using Mathematics and Computational Thinking
Student is asked to use mathematical reasoning or a calculation and interpret the results within the context of the given event, observation, or phenomenon.
1. Question gives an event, observation, or phenomenon.
2. Question asks student to perform a calculation or statistical test, generate a mathematical representation, or demonstrate a relationship between parameters.
3. Question asks student to give a consequence or an interpretation (not a restatement) in words, diagrams, symbols, or graphs of their results in the context of the given event, observation, or phenomenon.
33
Does this item elicit evidence of a �scientific practice?
Suppose that you have 10 grams of carbon and 4 grams of hydrogen available. Do you have enough starting materials to produce 14 grams of methane (CH4)? If yes, explain why and how much methane you can make. If no, explain why not and show the maximum amount of methane that could be prepared. Be sure to include a balanced chemical equation and all calculations needed to support your answer to receive full credit.
Using Mathematics and Computational Thinking
Student is asked to use mathematical reasoning or a calculation and interpret the results within the context of the given event, observation, or phenomenon.
1. Question gives an event, observation, or phenomenon.
2. Question asks student to perform a calculation or statistical test, generate a mathematical representation, or demonstrate a relationship between parameters.
3. Question asks student to give a consequence or an interpretation (not a restatement) in words, diagrams, symbols, or graphs of their results in the context of the given event, observation, or phenomenon.
✓
✓
✓
Item from Laverty et al. (2016)
34
The figure shows the concentration of atmospheric carbon dioxide from 1950 – 2008. Different color lines show predictions of the atmospheric carbon dioxide concentration in the future.
Based on the predicted concentrations of atmospheric carbon dioxide, which of the following models best predicts the pH of ocean water over time?
(A)
(B)
(C)
Data source: Intergovernmental Panel on Climate Change (2020)
Developing and Using Models
Student is given or asked to select a mathematical, graphical, computational, symbolic, or pictorial representation and select an appropriate explanation or prediction about an event, observation, or phenomenon based on the representation.
35
Developing and Using Models
Student is given or asked to select a mathematical, graphical, computational, symbolic, or pictorial representation and select an appropriate explanation or prediction about an event, observation, or phenomenon based on the representation.
The figure shows the concentration of atmospheric carbon dioxide from 1950 – 2008. Different color lines show predictions of the atmospheric carbon dioxide concentration in the future.
Based on the predicted concentrations of atmospheric carbon dioxide, which of the following models best predicts the pH of ocean water over time?
(A)
(B)
(C)
✓
✓
✓
x
Data source: Intergovernmental Panel on Climate Change (2020)
How can we address challenges of assessing practices?
36
Use data and phenomena that have already been curated
Use the 3D-LAP as guidelines for writing new items
Use the 3D-LAP to identify where existing items can be improved
Use closed-ended items to streamline grading
Assessing reasoning with multiple choice
37
Suppose that you have 10 grams of carbon and 4 grams of hydrogen available. Do you have enough starting materials to produce 14 grams of methane (CH4)?��a) Yes, because there is a total of 14 grams of carbon and hydrogen in the starting materials.�b) Yes, because producing methane only requires 3.5 moles of hydrogen and there are 4 moles of hydrogen in the starting materials.�c) No, because producing methane requires 0.875 moles of carbon but there are only 0.833 moles of carbon in the starting materials.�d) No, because the molar mass of 10 grams of carbon and 4 grams of hydrogen in the starting materials is less than the molar mass of methane.
Assessing reasoning with multiple choice
Which best explains the trend in the model you selected in the previous question?�
38
39
Assessing reasoning with dropdown items
Model [A/B/C] best shows how the pH of the ocean will change over time because carbon dioxide [dissolves into the ocean and forms an acid/is a greenhouse gas that warms both the atmosphere and the ocean].
40
Revisiting the anoles using selected response items�
b) Based on the mean survival rates, the researchers’ hypothesis
a) is supported because smaller lizards have a lower survival rate than larger lizards
b) is supported because larger lizards have a lower survival rate than smaller lizards
c) is not supported because the survival rate of smaller and larger lizards is approximately equal
d) is not supported because the smaller lizards have a higher survival rate than larger lizards
b) Based on the mean survival rates, the researchers’ hypothesis [is/is not] supported because the smaller lizards have [a higher/a lower/ the same] survival rate as the larger lizards.
c) Based on the mean survival rates, the average body size of anoles lizards is likely to
a) increase over time because larger lizards are more likely to survive predation
b) increase over time because smaller lizards have a higher mean survival rate
c) decrease over time because smaller lizards are more likely to be preyed upon
d) decrease over time because larger lizards have a higher mean survival rate
Plan ahead for practices�
41
Learning Objectives | | | |||||||
(List out learning objectives here) | | ||||||||
| Content�Knowledge | Asking Questions | Developing and Using Models | Planning Investigations | Analyzing and Interpreting Data | Using Mathematics and Computational Thinking | Constructing Explanations and Engaging in Argument from Evidence | Evaluating Information | Total |
LO1 | | | | | | | | | |
LO2 | | | | | | | | | |
LO3 | | | | | | | | | |
LO4 | | | | | | | | | |
Total | | | | | | | | | |
Plan ahead for practices�
42
Learning Objectives | | | |||||||
(List out learning objectives here) | | ||||||||
| Content Knowledge | Asking Questions | Developing and Using Models | Planning Investigations | Analyzing and Interpreting Data | Using Mathematics and Computational Thinking | Constructing Explanations and Engaging in Argument from Evidence | Evaluating Information | Total |
LO1 | 3 | | 2 | | | | 1 | | 6 |
LO2 | 3 | | 2 | 1 | | | | | 6 |
LO3 | 3 | | | | 1 | | | | 4 |
LO4 | 3 | | | | 1 | 2 | | | 6 |
Total | 12 | 0 | 4 | 1 | 2 | 2 | 1 | 0 | 22 |
Summary
44
Couch Lab: �Brian Couch�Mojtaba Khajeloo
Keenan Noyes
Dana Kirkwood-Watts
Karli Workman��
Acknowledgements
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. (DGE-1610400). Any opinion, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Anna Hiatt
Survey distribution:�David Bowne
Sam Donovan
Caitlin Hayes
Jaclyn Reeves-Pepin
cuminski2@unl.edu
@CrystalUminski
digital copy of the slides
Thank you!
Study participants
45
46