1 of 16

Assessment Critique�STAAR 5th Grade Science

Julie Black & Vanessa Manzano

2 of 16

3 of 16

  • STAAR is the state of Texas testing program that measures student achievement in reading, writing, mathematics, science, and social studies.  STAAR tests measure what students are learning, and whether they are ready for the next grade. STAAR is considered a summative assessment.

(TEA, 2022, All About the STAAR)

​

​

Description of Assessment

Click here for more information.

4 of 16

  • In grades 3-8, STAAR tests are given in the spring, and EOC tests are given at different times throughout the year. They are given to students at their own school and are administered in regular classrooms. 
  • STAAR is administered both on paper and online. 
  • After December 2022, all STAAR tests will be administered online.

(TEA, 2022, All About the STAAR)

​

​

Description of Administration

Click here for more information.

5 of 16

How did we conduct our Item Analysis?

​

  • First, we examined the items to determine how well they match the purpose and specifications of the assessment. ​

​

  • Using the Lead4ward data analysis, we found that all the items were aligned with the TEKS for 5th Grade Science.​

​

  • We also analyzed the multiple-choice answer data ​

​

  • Lead4ward provided information about each question and the TEK the question was to measure. We also used this tool for a detailed item analysis​

​

(2021-2022 Released Tests Aligned to the Standards, 2022)​

Click here for more information.

6 of 16

How did we conduct our Item Analysis?

    • Next, we examined the items for clarity and accessibility of language, free from complex linguistic construction.

​

    • We found concerns in items that included

​

      • Superfluous language 

​

      • Content that requires culture-specific background knowledge 

​

      • Problems with translation for Spanish speakers

​

​

​

7 of 16

Item Analysis

2021

Superfluous Language

(Alvarez et al, 2014, p.14)

8 of 16

Item Analysis

2021

Requires background knowledge that is not transferable across all cultures.

(Alvarez et al, 2014, p.14)

9 of 16

Item Analysis

Problems with Translation

2022

In Spanish, "electricity" is often referred to as "light." The lamp isn't working because there's no electricity often translates into Spanish as, la lampara no funciona porque no hay luz. (The lamp isn't working because there's no light). Students who use the word "light" instead of "electricity" would read answer choice A as, "Light energy and light energy."

10 of 16

How did we conduct our Item Analysis?

​

    • After that, we determined if the scoring criteria meet standards for validity and fairness.

​

    • We conclude that certain assessment items were not fair to all students, because of the three issues examined: 1. Superfluous language, 2. background knowledge required to answer an item correctly was not transferable across all cultures, and 3. Language difficulties that arise in translation.

​

11 of 16

How did we conduct our Item Analysis?

    • Finally, we looked at the technical quality of the directions and the format of both the assessment and response materials. 

​

​

    • We found that the online version of the exam had more accessible accommodations than the paper version, because eligible students can increase the size of the font, use text-to-speech, and hover over words for the definition. There is also a sticky note button that students can use as needed. 

​

(Yours in Education, 2022)

​

​

Yours in Education. (2022) 5TH GRADE SCIENCE STAAR 2021 RATIONALE

12 of 16

Reliability and Validity

    • The latest validity and reliability report as contracted by TEA was for the 2016 STAAR Grades 3-8 assessments
    • Findings by HumRRO supported the validity and reliability of the STAAR Science Grade 5 assessment scores
      • Majority of test items were aligned with TEKs expectations
      • Standard error of measurement was within an acceptable range
      • Test construction and scoring processes were consistent with industry standards.

​

​

(Independent Evaluation of the Validity and Reliability of STAAR Grades 3-8 Assessment Scores: Part 2 Final Report Texas Education Agency, 2016, pp.31-32)

13 of 16

Reliability and Validity

    • Reporting category 1 had one item rated as "not aligned"
    • Reporting category 2 was rated as "fully aligned"
    • Reporting categories 3 and 4 had one item rated as "partially aligned"

​

​

​

(Independent Evaluation of the Validity and Reliability of STAAR Grades 3-8 Assessment Scores: Part 2 Final Report Texas Education Agency, 2016, p. 32)

​

​

14 of 16

Suggestions for Improving the STAAR

​

    • Ensure questions only include content related information
      • Remove superfluous language
      • Questions should not have cultural bias
    • Replace the end-of-year assessment model to a “through-year” assessment model
    • STAAR revisions for the Spring 2023 administration will include question styles other than multiple-choice (Hudson, 2022)
      • Hot spots
      • Drag-and-drop
      • Multi-part
      • Multi-select
      • Short constructed response

​

Yours in Education. (2022) 5TH GRADE SCIENCE STAAR 2021 RATIONALE

15 of 16

STAAR 2021 Data

    • We created this table using data from Data Interaction for Texas Student Assessments
    • Why is the data for the subgroups not like the overall state data?

​

​

​

​

​

​

​

(Standard Summary - Spring 2021, Grade 5, STAAR, Science, n.d.)

16 of 16

References

2021-2022 Released Tests Aligned to the Standards. (2022). Lead4ward IQ Released Items Analysis Tool; lead4ward. https://lead4ward.com/iq/createiq.php

Alvarez, L., Ananda, S., Walqui, A., Sato, E., & Rabinowitz, S. (2014). Focusing formative assessment on the needs of English language learners. WestEd. 

Hudson, B. (2022, March 22). STAAR Redesign • Region 13’s Blog. Region 13’S Blog. https://blog.esc13.net/staar-redesign/

Independent evaluation of the validity and reliability of STAAR grades 3-8 assessment scores: Part 2 final report Texas Education Agency. (2016). Human Resources Research Organization. https://tea.texas.gov/sites/default/files/Independent%20Evaluation%20of%20the%20Validity%20and%20Reliability%20of %20STAAR%20Grades%203-8%20Assessments_Part2.pdf

Mahoney, K. (2017). The assessment of emergent bilinguals : supporting English language learners. Multilingual Matters.

Standard Summary - Spring 2021, Grade 5, STAAR, Science. (n.d.). Data Interaction for Texas Student Assessments; emetric. Retrieved September 10, 2022, from https://txreports.emetric.net/report?columns=4274%2C4276%2C4278%2C4279&crossdimension=false&dataids=2275&

disaggregate=bilingual%3Besl%3Blep&domain=1&filterdata=lep.0%3Blep.C&filterrelation=and&org=1&other=20&page= 1&report=40&sel=&summary=true&tgm=true&tgn=true&ts=large%7C0.0%3B2.0%3B3.0&type=table

Texas Education Agency. (2022). All About the STAAR Test. 

https://www.texasassessment.gov/staar-about

Yours in Education. (2022) 5TH GRADE SCIENCE STAAR 2021 RATIONALE. https://www.youtube.com/watch?v=gB44QOzCH6U&t=2447s&ab_channel=YoursinEducation