CCS Guide To Research & Program Evaluation

March 2025

We Are Family!

Examining Statistical Differences Between Constructed Response and Computer-Scorable Questions on an Accelerated Math Placement Test

Steven R. Hiner, Ed.D.

Introduction and Background

Columbus City Schools allows students to accelerate their learning in mathematics starting in Grade 7.

​

Identified students are given an assessment in March. Part of the assessment includes constructed-response questions.

​

Constructed-response questions were converted into computer-scorable questions in response to the COVID-19 pandemic.

​

This study compared the results of both assessments to determine the efficacy of continuing to offer the test completely online.

​

Objectives/Goals of Project

On a middle school math acceleration placement test, is there a significant statistical difference between constructed response and computer-scorable questions:

​

• Across the whole student population?
• When disaggregated by gender?
• When disaggregated by ethnicity?

​

Goal: To ultimately increase the number of students participating in acceleration by reaching as many students as possible in an equitable fashion.

Methodology: Population

Testing Qualifications

• Student is identified as Gifted/Talented in Math or Superior Cog
• Student is in enrolled in Math 6 during Grade 5
• Student earned a 4 or 5 on their Math 5 OST
• Student earned a grade of A in Math 5 during Grading Period 1 and at least a 3 on their Math 5 OST
• NWEA MAP Score: 230 or iReady Score: 500
• Teacher Recommendation

​

​

​

Methodology: Population

Methodology: Population

Methodology: Population

Methodology: Hypotheses

Research Question 1

On a math acceleration placement test, is there a significant statistical difference between constructed response and computer-scorable questions?

​

H₀: There is no significant statistical difference between constructed response and computer-scorable questions on a math placement assessment.

H_a: There is a significant statistical difference between constructed response and computer-scorable questions on a math placement assessment.

Methodology: Hypotheses

Research Question 2

On a math acceleration placement test, is there a significant statistical difference between constructed response and computer-scorable questions when disaggregated by gender?

​

H₀: There is no significant statistical difference between constructed response and computer-scorable questions on a math placement assessment when disaggregated by gender.

H_a: There is a significant statistical difference between constructed response and computer-scorable questions on a math placement assessment when disaggregated by gender.

Methodology: Hypotheses

Research Question 3

On a math acceleration placement test, is there a significant statistical difference between constructed response and computer-scorable questions when disaggregated by ethnicity?

​

H₀: There is no significant statistical difference between constructed response and computer-scorable questions on a math placement assessment when disaggregated by ethnicity.

H_a: There is a significant statistical difference between constructed response and computer-scorable questions on a math placement assessment when disaggregated by ethnicity.

Methodology: Data Collection

• Secondary Math Curriculum Coordinator already has test results for these students.

​

• Student OST scores were obtained through Infinite Campus.

​

• Student ethnicity and gender were obtained through Infinite Campus.

​

​

Methodology: Data Collection

In 2018, the middle school math placement assessment consisted of:

• MC items: 42 questions, 1 point each (72.4% of the total score)
• CR items: 5 questions, 2-4 points each (27.6% of the total score)

​

In 2021, the middle school math placement assessment consisted of:

• MC items: 42 questions, 1 point each (72.4% of the total score)
• Choice Matrix item: 1 question, 2 points (3.4% of total score)
• Match List item: 1 question, 4 points (6.8% of the total score)
• Cloze Math Formula: 3 questions, 2-4 points each (17.4% of total score)

Methodology

Methodology: Test Questions

Methodology: Test Questions

19

20

16

13

6

Findings

​

Results of All Students in the Population

​

• Each question was found to have statistically significant difference.
• Students performed better on the constructed response version of the test on four of the five questions.
• Students scored better on the computer-scorable version of Question 47 requiring students to type in their numerical answers.

19

20

16

13

6

Findings

​

A presentation of the evaluation results, organized around the research questions.

​

Results by Gender

​

​

• No significant statistical differences were found between genders on the two versions of the test.
• Means were within 1/100 of a point on two of the five questions when converted to computer scoring.

​

19

20

16

13

6

Findings

​

Results by Ethnicity

​

• Question 46 demonstrated significant statistical difference.
• This question showed significant interaction among between Hispanics / Latinos and Two or More Races.
• The CR version asked students to create a linear inequality and solve it for a given situation. The CS version had students drag and drop statements about linear inequalities into answer boxes.

19

20

16

13

6

Findings

​

• An analysis was performed to determine if there was a correlation between student scores on the math placement test and the Math 8 OST.
• The correlation for 2018 was 0.496 and for 2021 was 0.471.
• Correlations were determined to not have significant statistical difference.
• Based on similar correlations, the relationship between placement tests scores and Math 8 OST scores was not significantly different from 2018 compared to 2021. Students who were accepted into the program each year had similar Math 8 OST results.

​

Conclusion

While students scored slightly better on paper-based tests than computer scored tests, the pandemic impacted the results.

​

To ensure equity across all regions, the District should continue to offer the computer-scored version, testing all students in Math 6.

Recommendations

In 2006, Ohio Department of Education (ODE) stated:

“Students who can exceed the grade-level indicators and benchmarks set forth in the standards must be afforded the opportunity and be encouraged to do so. Students who are gifted may require special services or activities in order to fully develop their intellectual, creative, artistic and academic capabilities or to excel in a specific content area.”

​

The only way to ensure all district students are given the opportunity to accelerate is for the district to test all students in Grade 6.

​

​

Recommendations

• The Superintendent should direct the Curriculum Department to change the testing criteria, enabling all Math 6 students to test as recommended by ODE and the state.
• The Curriculum Director and Director of the Gifted & Talented program should continue to test online to free up district resources and allow for the increase in tested students.
• Students should be given time to practice for the test on the devices they will use during the test.
• Test directions should be updated to avoid confusion. Partial credit should be given where possible.

Lessons Learned

• Students will come across technology-enhanced questions as they take required state tests, including equation, gap match, grid, hot text, inline choice, matching, simulation, and table items (ODE, 2023). Research could be expanded by studying other technology-enhanced-items to add to the understanding of how different types affect student performance.
• Further studies could be performed on tests given to the general student population instead of possible accelerated students.
• Studies could be performed on other age groups and subjects, as this research focused on acceleration candidates taking Math 6.

Limitations

The major limitation of the study was the number of students participating in each year of the study.

• During the pandemic, students lost 35% of their learning with higher deficits in mathematics than reading (Betthäuser et al., 2022).
• In 2018, 153 Math 6 students were identified as gifted-talented, qualifying them to test. In 2021, only 66 gifted-talented students were identified.
• The number of students taking the accelerated math placement test dropped from 458 in 2018 to 341 in 2021.
• Students passing the test dropped from 2018 (41%) to 2021 (26%).

Delimitations

A major delimitation of the study was that the assessment used to compare testing types was for accelerated math placement of middle school students.

• Math was the only subject examined.
• Only students enrolled in Math 6 were studied.
• Students participating in the testing process had to be recommended for testing by either the curriculum department or their teacher, so the test was not open to all students.

Overall Take-Aways

​

​

All students in Math 6 should take the test.

​

Students should have time to practice on the device they will use to test.

​

District leaders should continue to examine the test and update it as necessary.

Thank you!

​

For questions, please reach out to researchproposal