Published using Google Docs
Product Efficacy Report: Relationship Between USATestprep Product Use and Student Performance
Updated automatically every 5 minutes

Relationship Between USATestprep Product Use

and Student Performance

A Product Efficacy Report

October 2019

By Christine Jax, Ph.D., Chief Academic Officer

Introduction

About USATestprep

USATestprep (USATP) is “by teachers, for teachers” and has been in business since 1998, with nearly two million students using its products each year. Recognized as a leader in K-12 in curriculum resources and test preparation, we offer a multitude of classroom resources, homework activities, and test review products by subject area and grade level. We help students perform in school, and we also prepare them for tests, such as end-of-course, end-of-grade, graduation, career readiness, and college entrance exams. Whether a teacher is looking for bell-ringers, in-class games, homework activities, instant remediation, or straight-up test-prep, we have the solution. Our programs are tailored to each state’s standards, whether home-grown or based on Common Core.

Commitment 

USATestprep supports, not supplants, teacher instruction. Products are online, automated, and created with optimal teacher flexibility in mind. Teachers can use virtually unlimited numbers of system-generated tests and assignments, or they can create their own supplemental resources and formative or summative assessments using our robust tools. We give teachers more time in the classroom to engage with their students and provide one-on-one instructional moments.

Importance of Efficacy

Efficacy is the ability to attain desired results. One of the results USATP seeks is improved test scores from schools where students use USATestprep products. From a research standpoint, this also helps ensure criterion validity. This study is part of a series of studies investigating the efficacy of our products.

What Was Studied

This product efficacy study on product use as it relates to state assessment performance  investigated the relationship between the high use of USATestprep products at particular schools and the performance of the students at those schools on state assessments.  In particular, the study 1) compared the average score on state assessments of students at the schools in the sample to the average state-wide score, and 2) compared the average score on state assessments of students at the sample schools in 2018 to those in 2019. Sixteen schools were studied, four from each of Florida, Georgia, Mississippi, and South Carolina. The state assessments used for this study were English language arts and math for sixth and eighth grade in all states, with the addition of seventh grade English language arts and math in Georgia, Mississippi, and South Carolina; the addition of eighth grade science in Florida and Georgia; and the addition of eighth grade social studies in Georgia.

Main Findings

The results indicate schools that have high usage of USATestprep products have student test scores well above the state average and see improvement in test scores year over year. Across all four states:

The Study

Background

The primary reason teachers use USATestprep products is to help prepare their students for state assessments. Consequently, teachers want assurance that students who use USATestprep products will perform well on the state assessments. This study was designed to demonstrate the effectiveness of USATestprep products and to establish criterion validity.

The two objectives of the study were to determine if

  1. High use of USATestprep products led to a higher percentage of students performing well on state assessments compared to the average statewide performance.
  2. High use of USATestprep products led to better performance year over year on state assessments.

Design

This was a retrospective non-cohort quantitative study looking at the relationship between the use of USATestprep products at particular schools and the performance on state assessments of students from those schools.

In order to have representation from more than one state, but to limit confounding variables, the Southeastern region was selected, and the top three states in terms of usage of USATestprep products were selected within that region. Those states were Florida, Georgia, and South Carolina. High usage was determined based on the number of products being used.

Next, the top four schools within each state were identified, in terms of product usage. Usage was defined as number of teacher logins. From those schools, grade levels and subject areas to compare were selected based on two criteria: 1) the state required an end-of-grade assessment for particular grades and subject areas; and 2) the school used USATestprep products for some of those subjects. Those resulting grade levels and subject areas were English language arts and math in grades six and eight in all states. The top usage schools in Florida and Georgia also met the two criteria in English language arts and math in grade seven; Florida and Georgia met the two criteria for science in grade eight; and Georgia met the criteria for social studies in grade eight.

Sampling

The sampling method was non-probability purposive convenience sampling. The sample consisted of twelve schools, four from each of three states in the Southeastern region of the United States. The states were Florida, Georgia, and South Carolina. The states were selected based on their high use of USATestprep products to ensure high-use schools would be available. The schools were selected from each state based on the following criteria: 1) the school had used USATestprep products prior to state testing in 2018 and 2019; 2) the school was identified as a high-use school, based on the number of logins compared to other schools; 3) the school used USATestprep middle school math and ELA products; 4) the school was a public school whose students were required to participate in state math and ELA end-of-grade assessments in grades six, seven, and eight.        

Description of sample

The schools are labeled by the two-letter state designation and then a number:

FL-1        Grades: 6 -8; Students: > 2,000; Hispanic: 60%; White: 26%; Asian: 8%; Black

3%; ELL: 8%; Low-income: 19%; Student-teacher ratio: 24:1

FL-2        Grades: 6-8; Students: Between 1,000 - 2,000; Hispanic: 39%; White: 35%;

Black: 19%; Asian: 4%; ELL: 8%; Low-income: 63%; Student-teacher ratio 31:1

FL-3        Grades: 6-8; Students: Between 1,000 - 2,000;  White: 50%; Hispanic: 38%

Black: 6% Black; Asian: 2%; ELL: 3%; Low-income 94%; Student-teacher ratio

19:1

FL-4        Grades: 6-8; Students: Between 1,000 - 2,000; White: 50%; Hispanic: 35%;

Black 3%; Asian: 2%; ELL: 3% Low-income: 52%; Student-teacher ratio 18:1

GA-1        Grades: 6-8; Students: Between 1,000 - 2,000; White: 37%; Black: 30%;

Hispanic: 26%; Asian: 3%; ELL: 9%; Low-income: 40%; Student-teacher ratio

16:1

GA-2        Grades: 6-8; Students: < 1000; White: 47%; Black: 29%; Hispanic: 13%; Asian:

4%; American Indian/Alaska Native: 1%; ELL: 3%; Low-income: 45%;

Student-teacher ratio 16:1

GA-3        Grades: 6-8; Students: Between 1,000 - 2,000; White: 71%; Hispanic: 18%;

Black: 6%; Asian: 2%; ELL: 4%; Low-income: 37%; Student-teacher ratio 16:1

GA-4        Grades: K-12; Students: < 1000; White: 52%: Black 33%; Hispanic: 7%; Asian:

2%; ELL: 0; Low-income: 45%; Student-teacher ratio 37:1

SC-1        Grades: 6-8; Students: Between 1,000 - 2,000; White: 46%; Hispanic: 35%;

Black 15%; Asian: 1%; ELL: unknown; Low-income: 49%;  Student-teacher

ratio: 17:1

SC-2        Grades:6-8; Students: < 1000; White: 85%; Hispanic: 6%; Black: 5%; American

Indian/Alaskan Native: 1%; ELL: unknown; Low-income: 58%; student-teacher ratio

17:1

SC-3        Grades: 6-8; Students between 1,000 - 2,0001,089; White: 60%; Black: 25%;

Hispanic: 8%; Asian: 1%; American indian/Alaska Native: 1%; ELL: unknown;

Low-income: 60%; Student-teacher ratio: 14:1

SC-4        Grades: 6-8; Students between 1,000 - 2,000; White: 45%; Black 24%;

Hispanic: 22%; ELL: unknown; Low-income: 100%; Student-teacher ratio: 17:1

Data Collection and Analysis

Data regarding the product usage for the twelve schools was obtained through an automated report through the USATestprep platform. Data regarding student performance on state assessments was obtained from publicly reported data through each state’s website.

The objectives of the study were to determine if high use of USATestprep products led to a higher percentage of students performing well on state assessments, and if high use of USATestprep products led to better performance year over year on state assessments.

Limitations

Findings

The results indicate schools that have high usage of USATestprep products have student test scores well over the state average and see improvement in test scores year over year. This was the case among all grades and subject areas investigated.

Across all three states, students at schools with high use of USATestprep products for test preparation out-performed the state mean by an average of 10.42%. The range was 6.8% - 14.45%. Looking only at the shared courses among the states (grades 6 and 8 ELA and math), students at schools with high-use of USATestprep products out-performed the state mean by an average of 10.13%.

Analysis of the data of pass rates by state showed Florida USATP schools performing well above the state average at 14.5% across all subjects and grade levels, with a considerable difference in eighth grade math where the students in the USATP high-use schools had a pass rate 31.5% higher than the state average. South Carolina USATP schools performed 10% above the state average for all subjects and grade levels combined, with the most notable subject being sixth grade math at a pass rate almost 17% above the state average. The results for Georgia USATP schools, at 6.8% higher than the state average pass rate across subjects and grades, were somewhat lower than for Florida and South Carolina, although for eighth grade science the pass rate was 14% higher.

                

There was a 3.23% average improvement in school test scores from 2018 to 2019 on assessments where USATestprep products were highly used in preparation for the state assessments. The range was 2.4% - 3.75%. Looking only at the shared courses among the states (grades 6 and 8 ELA and math), the average improvement in school test scores from 2018 to 2019 was 3.21%. There was no pattern among the states in year-over-year improvement levels.. There were three incidences where there was no improvement from 2018 to 2019: eighth grade science in Florida, sixth grade math in Georgia, and eighth grade math in South Carolina. The comparison of scores from 2018 to 2019 is a potential measure of teaching rather than learning, as there are different students taking the tests in the two years.

           

Conclusion

The findings indicate that high use of USATestprep products corresponds to high student performance on state assessments in all twelve schools and across all subject areas. This likely is attributable to the increase in student knowledge and comfort with the standards as a result of using USATestprep materials, since this is substantiated by the results of the first efficacy study on teacher satisfaction wherein over 95% of teachers reported that USATestprep improved their students’ achievement and standards mastery.

Comparison of 2018 and 2019 pass rates at USATP high-use schools suggests continued use of USATestprep products increases pass rates year over year. This is a measure of teacher and/or school effectiveness in teaching to the standards and likely is the result of improved teacher familiarity and competence with USATP products and product features (Kini  & Podolsky, 2016).


References

Allensworth, E. M., Farrington, C. A., Gordon, M. F., Johnson, D, W., Klein, K., McDaniel, B., &

Nagaoka, J. (2018). Supporting social, emotional, & academic development: Research

implications for educators. Chicago, IL: University of Chicago Consortium on School

Research.

Ferguson, H.B., Bovaird, S., Mueller, M.P. (2007). The impact of poverty on educational

outcomes for children, Paediatrics & Child Health, Volume 12, Issue 8, October 2007,

Pages 701–706, https://doi.org/10.1093/pch/12.8.701

Kini, T., & Podolsky, A. Does Teaching Experience Increase Teacher Effectiveness? A Review

of the Research (Palo Alto: Learning Policy Institute, 2016).

Krejcie, R. V., & Morgan, D. W. (1970). Determining sample size for research activities.

Educational and Psychological Measurement, 30, 607-610.

Morse, J. M. (1994). Designing qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.),

Handbook of qualitative inquiry (pp. 220-235). Thousand Oaks, CA: Sage.

Reichheld, F. (2003). The one number you need to grow. Harvard Business Review,

December, 2003. Brighton, MA: Harvard Business Publishing.

Vanderhaar, J.E., Muñoz, M.A. & Rodosky, R.J. (2006). Leadership as Accountability for

Learning: The Effects of School Poverty, Teacher Experience, Previous Achievement, and Principal Preparation Programs on Student Achievement. Journal of Pers Eval Educ (2006) 19: 17. https://doi.org/10.1007/s11092-007-9033-8


Appendix

Student performance: Average percentage over the state average for the sample population.

FLORIDA

GEORGIA

SOUTH CAROLINA

6th grade ELA

8.75

4.50

8.00

6th grade math

5.50

7.00

16.75

7th grade ELA

6.25

8.50

7th grade math

3.25

15.00

8th grade ELA

14.50

6.75

5.00

8th grade math

31.50

4.75

8.75

8th grade science

12.00

14.00

8th grade social studies

8.00

Student performance: Average percentage increase year over year for the sample population

FLORIDA

GEORGIA

SOUTH CAROLINA

6th grade ELA

.50

7.75

1.25

6th grade math

6.50

-.50

4.25

7th grade ELA

1.25

6.50

7th grade math

.50

5.00

8th grade ELA

1.25

4.50

6.50

8th grade math

10

0

-.50

8th grade science

-.05

2.75

8th grade social studies

8.00