Improving

Ranking Arkansas High Schools�by Performance vs Predicted Performance

2022-2023

1

Table of Contents

1. Executive Summary p. 3 – 4
2. Introduction p. 5 – 9
3. Methodology p. 10 – 31
1. Data Collection p. 10 – 11
2. Data Cleaning p. 12 – 15
3. Exploratory Data Analysis p. 16 – 18
1. K-Means Clustering Model p. 16 – 18
2. Scatter Plots p. 19 – 28
3. Prediction/Ranking Models p. 29 – 31
4. Results p. 32 – 35
5. Conclusion p. 36 – 39
6. Appendix p. 40 – 41

​

​

​

2

Executive Summary

• This project utilizes a regression model constructed with demographic data as inputs to predict a school’s average ACT Composite score, then ranks Arkansas high schools according to how they performed against the model’prediction. ��Most ranking systems rank schools according to how well their students perform on standardized tests. Schools with higher average scores rank higher than those with lower scores. But many factors that impact student outcomes are beyond control of the schools themselves. ��The rankings in this report are not intended to encourage parents to send their children to this school or that school, but to rank schools according to how well they outperform or underperform against expectations determined by the mathematical model.

3

Executive Summary

• The final rankings determined by the model are surprising in some cases. Some schools with extremely low average ACT Composite scores perform relatively well against expectations predicted by the model. Other schools’ rankings are not surprising. ��The model developed in this report predicts school performance based on input variables—demographic characteristics—that are beyond the control of the schools. Therefore, schools that beat their predicted ACT Composite averages by 2 points, rank higher than schools that beat their prediction by fewer, even if the average ACT Composite is higher for the lower ranked school.��Schools that beat predictions should be studied further, as should those schools that underperform. Future studies that follow from this report should seek to determine what characteristics the schools CAN control that are associated with over or underperformance. ��

4

Introduction

• Most school ranking systems report to parents, politicians, and policymakers which schools offer students the best education, and which offer the worst education. They create a final score based on criteria weighted according to the importance the people who do the ranking place on specific criteria. Test scores, diversity, curriculum, and/or other variables are assigned a metric and a weight and incorporated into the model. This project creates a different kind of ranking system. ��The rankings produced by this report judge schools based on how well they beat or miss expectations, based on a multiple linear regression model that predicts a school’s average ACT Composite score. A school that beats its predicted score by 2 points will rank higher than a school that beats its predicted score by 1.5 points, even if the lower ranked school has a higher average ACT Composite score.

5

Introduction

• The average ACT Composite score provides a good measure of the overall education schools offer their students because all 11^th grade students in Arkansas have been required to take the ACT for the past several years. It is a common assessment designed to gauge a student’s college readiness that is administered to virtually all Arkansas students. ��However, to rank schools by their average score is not a fair measure of how well schools are serving their students. Demographic characteristics known to influence test scores vary greatly from one school to another. Poverty, enrollment, and race are all variables that influence a school’s test scores, among others. School’s do not control these variables, but their influence on test scores is undeniable.

6

Introduction

• “Poverty has become one of the most prevalent indicators of academic achievement in our schools today.”^2��“That is, underserved minority students were more likely to have lower ACT scores than white students;”³��The regression model essentially “averages” the performances of the schools with consideration for the values of the input variables for each school in the rankings. An estimate is produced for each school based on its values for each of the input variables. The assumption is that, if two schools share the same characteristics in all variables other than the input variables used in the model and share the same values for all the input variables, then we would expect essentially the same performance on standardized test scores.�

7

Introduction

• The goal of all educators and all educational institutions should be to provide the very best education possible to the parents and students served by the school. If two schools have similar demographics that influence test scores and can not be controlled by the school, and one of those schools is significantly outperforming the other, everyone should want to know what is different between the two schools. ��While this project does not examine what higher ranking schools are doing to outperform their peers, it does lay the groundwork for future research to do just that. Some schools that fly under the radar in traditional ranking systems are performing quite well considering the demographics of their student body, even if their average ACT Composite score is relatively low.

8

Introduction

• The rankings that result from this report are not intended for parents to make decisions on where to move, what schools to enroll their children in, or other goals typical of most school ranking systems. ��This goal of this project is to encourage educators to learn why some schools outperform expectations, and hopefully make changes to the way underperforming schools operate, thereby improving the educational outcomes of students in those schools.

9

Methodology

• Data Collection��Data was retrieved from the Arkansas Department of Education website³ for Arkansas school report cards. Data requested was chosen from what was available on the ADE website and because it was believed it could impact the average ACT Aspire Composite score. ��The average ACT Aspire Composite was chosen as the response variable for several reasons. First, the ACT is a college entrance exam designed to measure a student’s preparedness to succeed in post-secondary education. Second, the state of Arkansas has mandated students take the ACT in a statewide sitting in the spring semester of their 11^th grade year. Therefore, it provides a reasonable for measuring the average education a student can expect to receive at a particular school.

​

​

10

Methodology

• Data Collection��The purpose of this project is to predict average ACT Composite scores using variables over which the schools have little or no control so those schools can be ranked by their performance against an expectation. For this reason, variables that could impact educational outcomes but could not be controlled by the school were omitted as inputs.��Other variables that would only affect a small number of schools were also omitted. For instance, only race data for white, black, and Hispanic students was collected. Populations of other races were deemed small enough in most schools to be omitted.

11

Methodology

• Data Cleaning��The data set downloaded from ADE required some manipulation to make it suitable for this project. First, the data set included all schools. After the CSV file was uploaded as a data frame, all schools that did not serve grade 12 in the 2021-2022 school year were eliminated. ��The CSV file automatically converted the grade bands served by the schools to dates in the spreadsheet. The value 9-12 in the data set was converted to September 12, 2023. Several attempts to reformat the cells failed. These were then converted to string variables in the form “9-12”, “10-12”, etc. (parentheses included) to avoid the automatic conversion to dates. Once uploaded as a data frame, these strings were converted to integers that represented the number of grades served. “9-12” was converted to 4, “10-12” to 3, etc.

12

Methodology

• Data Cleaning��Null values in the data set were present for schools without a particular trait. For instance, a school with 0 black students contained a null value for the proportion of black students. All null values were converted to 0 so the model could evaluate them. ��The data set contained “RV” entries, for “restricted value.” These were entered for data that represented fewer than 10 students at a school. The “RV” values appeared where numeric values existed, but ADE believes the data could reveal information about specific students since it applied to so few. The “RV” values were replaced with the mean of their column.�

13

Methodology

• Data Cleaning��The first model created was a K-Means Clustering model, to create groups of schools with similar characteristics that might not be identified by the authors. Box plots were created for each cluster to show the distribution of average ACT Composite scores for the 2021-2022 school year, revealing several outliers.��It was determined some of the outliers should be removed from the data frame because they were so different from most schools. One of the schools removed is Haas Hall Academy, the top-ranked Arkansas high school in most school ranking systems. HHA is a charter school. It was decided that many of the requirements HHA places on its students and parents are not transferrable to typical Arkansas high schools, so it was removed. Schools that graduated fewer than 15 students in the 2021-2022 school year were also eliminated from the dataset. These included several schools administered by the Arkansas Department of Human Services for juveniles in the custody of the state for correctional purposes. It was decided these schools were atypical enough to remove them from the dataset also.��After removing certain outliers and special cases, 232 high schools were used to create the model.

14

Methodology

• Data Cleaning��One data element to be examined was 2021-2022 enrollment. However, because schools in the dataset included some serving 3 grades and some serving as many as 8 grades, enrollment as a variable required standardization. This was achieved by creating a new variable called enrollment ratio, dividing total number enrolled by the number of grades served by the school. ��Enrollment ratio yields the mean number of students per grade for each school. �

15

Methodology

• Exploratory Data Analysis��First, a K-Means Clustering model sorted the schools into three groups. A total of 239 variables were incorporated. Demographic data, teacher qualification data, ACT Aspire data for grades 9 and 10, and enrollment in AP course data, all for school years 2013-2014 to 2021-2022. If data was available for the 2022-2023 school year, that data was included as well. ��This K-Means Clustering algorithm groups schools based on commonalities in the variables considered to build the model. These common factors are not always recognizable, especially with so many variables considered. If the distributions of ACT Composite scores differ from group to group, there is reason to believe at least some of the variables considered influence ACT Composite scores. �

16

Methodology

• Exploratory Data Analysis��A K-Means Clustering Model was created to divide the schools into 3 groups based on similar characteristics. Group 1 consisted of 13 schools, Group 2 contained 26 schools, and there were 193 schools in Group 3.�

17

Methodology

• Exploratory Data Analysis��The differences in the distributions of ACT Composite scores among the three groups suggested that further investigation is warranted.��Since a goal of this project is to create a model that will predict ACT Composite scores based on variables uncontrollable by the schools, the next task was to identify such variables that potentially affect the schools scores. ��To accomplish this, scatterplots were created, plotting the individual variable versus the ACT Composite scores for each of school. Variables that appeared to have no association with the model were not included to build the prediction model. ��The different groups identified by the K-Means Clustering model were plotted as different colors to help identify trends within the various groups and trends that varied from group to group. ��Group 1 – magenta, Group 2 – blue, Group 3 – red.

18

Methodology

• Exploratory Data Analysis��The enrollment ratio revealed a positive correlation with a school’s 2021-2022 average ACT Composite score. ��This graph clearly showed the groups from the K-Means Clustering model were separated largely by enrollment.��Magenta = Group 1�Blue = Group 2�Red = Group 3�

19

Methodology

• Exploratory Data Analysis��The number of grades served showed a possible negative correlation with a school’s 2021-2022 average ACT Composite score, even though there is significant variability in similarly structured schools. ��Magenta = Group 1�Blue = Group 2�Red = Group 3�

20

Methodology

• Exploratory Data Analysis��We see a negative correlation between the proportion of student body that is black and a school’s 2021-2022 average ACT Composite score. ��The racial disparity in standardized test scores is well documented. The trend here is not surprising, but still disturbing.��Magenta = Group 1�Blue = Group 2�Red = Group 3�

21

Methodology

• Exploratory Data Analysis��This was somewhat surprising. There appears to be no association between the proportion of Hispanic students in a school and its scores. ��It is known the racial disparity in ACT scores extends to Hispanic students⁴ as well. The 2021-2022 ACT scores saw Hispanic students in Arkansas average 2.4 points lower than their white counterparts⁵. But the plot clearly shows no real association between the proportion of Hispanic students and the ACT Composite average for the school.��Magenta = Group 1�Blue = Group 2�Red = Group 3�

22

Methodology

• Exploratory Data Analysis��There is a clear, positive association between the proportion of white students and the 2021-2022 ACT Composite Scores. Again, the racial disparity in test scores confirmed.����Magenta = Group 1�Blue = Group 2�Red = Group 3�

23

Methodology

• Exploratory Data Analysis��A 504 plan is intended to remove barriers for students with disabilities, including learning disabilities. The plot suggests that the ACT score increases as the proportion of the student population with 504 plans increases. ��Magenta = Group 1�Blue = Group 2�Red = Group 3�

24

Methodology

• Exploratory Data Analysis��The relationship looks very similar to the earlier graph showing the relationship between the proportion of Hispanic students and the school’s scores. There is no clear association.��Only a handful of schools in Groups 1 and 3 have proportions of ELL greater than 20%. No schools in Group 2 have proportions of ELL greater than 20%.��Magenta = Group 1�Blue = Group 2�Red = Group 3�

25

Methodology

• Exploratory Data Analysis��There is a negative correlation between the proportion of SPED students and the average ACT Composite score.��Magenta = Group 1�Blue = Group 2�Red = Group 3�

26

Methodology

• Exploratory Data Analysis��The proportion of students with dyslexia does not appear to be associated with ACT Composite scores. ��Magenta = Group 1�Blue = Group 2�Red = Group 3�

27

Methodology

• Exploratory Data Analysis��The proportion of students on the free and reduced lunch program is often used to measure poverty among the student body. Here we see an undoubtable association between poverty and scores. ��Magenta = Group 1�Blue = Group 2�Red = Group 3�

28

Methodology

• Exploratory Data Analysis��The following input variables were chosen to build a regression model to predict a school’s average 2021-2022 ACT Composite score.
• Grades served by school.
• Enrollment ratio (enrollment/number of grades served.)
• Proportion of students on free/reduced lunch program.
• Proportion of student body that is black.
• Proportion of student body that is white.
• Proportion of student body that have 504s.
• Proportion of student body receiving SPED services.�

29

Methodology

30

Methodology

• Ranking Model��The predictive model developed to predict a school’s average ACT Composite score reveals an expected score based on each school’s demographic data considered in the prediction. Each school’s actual 2021-2022 average ACT Composite score is then compared to its prediction, actual – prediction, yielding a measure of how much better or worse than the prediction the school performed.��2021-2022 Residual = 2021-2022 Average ACT Composite – 2021-2022 Predicted ACT Composite��Schools are then ranked in order of their residual. The greater the residual, the better the school performed compared to its prediction. ��A school with an actual average ACT Composite score of 16 and a predicted score of 14 will rank higher than a school with an actual score of 20 and predicted score of 19.�

31

Results

• Recall, Haas Hall Academy and several other schools were removed from the dataset before building the model. HHA was an outlier on the high end and the others were outliers on the low end that graduated fewer than 15 students in the 2021-2022 school year. Another school that serves only a few disaffected youth in central Arkansas was also removed when it revealed itself as an outlier when the proportion of SPED students was plotted against average ACT Composite scores.��There were 232 Arkansas high schools ranked by the model after removing outliers and special cases. Of those, 111 high schools topped their predicted scores and 121 scored below their predicted scores.

32

Results

• A plot of residuals against the proportion of 2021-2022 students on Free and Reduced Lunch in each school is shown. Similar plots were created for each of the explanatory variables. None revealed patterns to suggest their use in the regression model is inappropriate. These can all be viewed on the Jupyter Notebook linked in the appendix.

33

Results

• A histogram reveals the shape of the distribution of residuals is approximately normal, again suggesting the regression model is appropriate.

34

Results

• The 10 highest ranked schools are shown here, along with their residuals. Recall the residual is calculated by subtracting the school’s predicted average ACT Composite score from its actual average ACT Composite score in the 2021-2022 school year. The complete list of all ranked schools can be viewed here.

35

Conclusion

• Once again, this is not a typical rating system. It is not intended to help parents choose which school their children should attend. It is not intended for use by college admission agents to determine if students received an education that prepared them for success in college. A school ranked number n should not be considered a “better school” than the school ranked number n + 1.��Instead of ranking schools based on how well their students performed on a test or tests, this system ranks them based on how well their students missed or exceeded a predicted score based on the demographics of the student body.

36

Conclusion

• The project considered data from 232 of 244 Arkansas high schools to build the prediction model. Schools removed include outliers and special cases that make them not representative of typical high schools in Arkansas.��All other things being equal, high schools that share common values of the explanatory variables incorporated in the model should see similar performance on the ACT. Significant differences in performance suggest there are other variables at play that cause one school to outperform another. ��Policymakers and politicians should also consider variables beyond the control of schools, but within the control of communities and local governments. Opportunities to improve educational outcomes may lie in the hands of entities outside the halls of the schools.

37

Conclusion

• This system is not perfect. Some schools may rank high or low due to variables the model did not take into account. For instance, Springdale High School ranked very low. Springdale has a significant proportion of Marshallese students. The population of Marshallese students in most districts in the state is at or near 0. It could be the ethnic makeup of Springdale High School explains its relatively low average ACT Composite score, but its Marshallese population is not accounted for by the model. ��Other variables may exist that positively or negatively influence a school’s performance that are not accounted for in the model. When evaluating a particular school’s ranking, this should be considered. ��However, in the absence of such variables, schools ranked high by the model should be examined to determine what they do differently that can be controlled in each school.

38

Conclusion

• This system is not perfect. Some schools may rank high or low due to variables the model did not take into account. For instance, Springdale High School ranked very low. Springdale has a significant proportion of Marshallese students. The population of Marshallese students in most districts in the state is at or near 0. It could be the ethnic makeup of Springdale High School explains its relatively low average ACT Composite score, but its Marshallese population is not accounted for by the model. ��Other variables may exist that positively or negatively influence a school’s performance that are not accounted for in the model. When evaluating a particular school’s ranking, this should be considered. ��However, in the absence of such variables, schools ranked high by the model should be examined to determine what they do differently that can be controlled in each school.

39

Appendix

Sources��¹ The Effects of Poverty on Academic Achievement, Kendra McKenzie, https://files.eric.ed.gov/fulltext/EJ1230212.pdf�² Arkansas Department of Education Data Center,

https://myschoolinfo.arkansas.gov/Plus/Schools

³ Adjusted differences in ACT Scores by Race/Ethnicity; Daniel M. McNeish, Justine Radunzel PhD, Edgar Sanchez PhD,

https://www.act.org/content/dam/act/unsecured/documents/5692-data-byte-2016-7-adjusted-differences-in-act-scores-by-race.pdf

⁴ E(race)ing Inequities|How Does Race Influence ACT Scores?; James E. Ford and Nicholas� Triplett, � https://www.ednc.org/eraceing-inequities-how-does-race-influence-act-scores/

​

⁵ 2021-2022 ACT Profile Report – Arkansas; Arkansas Department of Education� https://dese.ade.arkansas.gov/admin/Files/049999_Arkansas_PROFILE-RPT-STATE-11_LS.pdf

40

Appendix

Code

Jupyter Notebook with Code

Rankings

Complete Rankings��

41