Wrestling with Undergraduate Grades

at a Large, Public University

​

August (Gus) Evrard¹, Cait Hayward², Rebecca Matz²,

¹Departments of Physics and Astronomy, ²Center for Academic Innovation

University of Michigan

​

2

Who are we?

https://ai.umich.edu/who-are-we/our-team/#r_and_d

Associate Director for

Research & Development,

Center for Academic Innovation

Data Scientist,

Center for Academic Innovation

Professor, Physics & Astronomy

Architect & Evangelist,

Center for Academic Innovation

Research Scientist,

Center for Academic Innovation

The Center for Academic Innovation is responsible for initiatives that aim to reimagine the future of higher education and support both teachers and learners through curricular and technological innovations.

We do this through three pillars of work:

• Online learning experiences (MOOCs, Teach-Outs, hybrid and online degrees)
• Educational technologies (Atlas, ECoach, GradeCraft, Gamut)
• Educational research

3

4

Grades reflect:

• Engagement
• Proficiency
• Compliance
• Ranking

They are also:

• Changing over time
• Reflective of systemic inequities
• Opportunities to engage conversations around curricular practice

5

Who are we?

LAK 2017 practitioners track

Evrard + Teplovs: Community Building Around a Shared History: Rebooting Academic Reporting Tools at the University of Michigan

Topic 1:

​

The Role of Selectivity in Grade Inflation

at a Large, Public University

​

August (Gus) Evrard¹, Cait Hayward², Kyle Schulz²,

¹Departments of Physics and Astronomy, ²Center for Academic Innovation

University of Michigan

​

LAK21 proceedings

(best short paper nominee)

https://dl.acm.org/doi/abs/10.1145/3448139.3448199

​

7

What is grade inflation?

What wikipedia says -

​

Grade inflation (also known as grading leniency) is the awarding of higher grades than students deserve, which yields a higher average grade given to students.^[1]

The term is also used to describe the tendency to award progressively higher academic grades for work that would have received lower grades in the past. However, this is not grade inflation, as higher grades in themselves do not prove grade inflation and many^[who?] believe there is no such problem. For this to be grade inflation, It is necessary to demonstrate that the grades are not deserved.^[1]

https://en.wikipedia.org/wiki/Grade_inflation

Our practical perspective -

​

Grade inflation is a rise in average grade earned by students completing courses within a subject or set of subjects that cannot be attributed to an improvement in student aptitude. i.e., A mean grade rise attributable to an institution’s faculty.

8

Grade inflation is certainly not a new topic…

9

But has grade inflation reached its nadir?

And still grades continue to rise at many institutions… yawn!

10

Yale has initiated its own discussion about grading policies in the last year, forming an ad hoc committee on the subject. In a review last spring, that committee found that 62 percent of grades awarded at Yale College from 2010 to 2012 were in the A-range.

Michigan is not immune...

11

c.f., ART2.0 grade landscape poster , 2017 U-M Provost’s Seminar on Teaching, Beyond Grades, Center for Research on Learning & Teaching

2018:

61% A-level grades

12

Practical definition: employ ACT/SAT as a measure of aptitude

Our practical perspective -

​

Grade inflation is a rise in average grade earned by students completing courses within a subject or set of subjects that cannot be attributed to an improvement in student aptitude.

Historically, standardized test scores are a mainstay measure of student aptitude. The ACT* and SAT* in the US have been widely employed by university admissions offices as a key indicator of student quality. �In addition, the 4.0 grade point scale (A=4, B=3, etc.) is ubiquitous at US universities.

For our historical study, we use SAT/ACT score as a proxy for student quality.

​

​

An increasingly selective institution is one at which the SAT/ACT scores of the student population are increasing over time.

* formerly “American College Test” and “Scholastic Aptitude/Assessment Test”; names are no longer acronyms.

Core Q: How does degree of selectivity affect grades earned?

​

We analyze student data for a public U that became increasingly selective, in the sense of rising mean ACT and SAT scores of incoming undergraduates, over the years 2006 to 2019.

13

“big” data, ~0.5M students and >3M grades, offer good statistical precision,

but only one university!

​

Mean SAT and ACT scores (cols. 4, 7) rise steadily each year.

Rank correlations of SAT/ACT score with GPA are persistent

14

R² ~ 0.1, “small effect”.�~90% of grade variance is due to other sources.

But the correlation is persistently positive.

study period

study period

Model basis

​

Thought experiment:

Imagine that we could freeze the curriculum for several years while admission practices drift over time.

We’d expect grades on campus to vary with student quality, going up if higher-scoring students are admitted and down if the opposite.

​

We exploit conditional statistics to effectively run this experiment.

15

Actual grades

​

​

16

Similar shifts in mean grade are seen for all credit hour levels.

Mathematical model: grade susceptibility to test scores

The joint probability density function (PDF) of grades and pre-college scores, Pr(g,s), can be expressed as the product of the conditional probability density, Pr(g|s), of grades at a given score and the PDF of pre-college scores, Pr(s).

​

​

​

This equation is true by construction for any given year, t.

​

17

We call the conditional PDF, Pr(g|s,t), the grade susceptibility to test score, and interpret it as a practical indicator of academic rigor.

​

Under this interpretation, a rise in grades conditioned on student test scores constitutes grade inflation attributed to an institution’s faculty.

Mathematical model: predicting how selectivity affects grades

Our model for selectivity’s influence is based on freezing the conditional probability at some chosen (reference) year, then developing model expectations for grades using

​

​

The model mean grade is computed by Monte Carlo realization of virtual student samples,

​

​

​

​

​

​

18

We analyze different academic domains, D :

– entire university,

– business (BUS), engineering (ENG), and liberal arts & sciences (LAS) colleges,

– divisions of humanities (HUM), social (SOC) and natural (NAT) sciences within LAS.

Result for all undergraduate courses

19

Selectivity accounts for nearly HALF of the overall rise in mean undergraduate grades.

​

​

Does this vary by academic domain or division?

​

Teal shows actual grades

Brown shows model expectations (w/ bootstrap error)

Results for each academic domain

20

Liberal arts & sciences Business Engineering

Teal shows actual grades

Brown shows model expectations (w/ bootstrap error)

~20 years to 4.0 average

Revealing faculty-related degrees of grade inflation

21

Faculty component

(4.0 scale)

low rates in business and engineering

​

​

higher in humanities and social science

​

natural sciences > engineering?

(4 sigma significant)

What role might budget model / market forces play?

22

# English majors is dropping

# CompSci majors is expanding

Summary

• We introduce an importance sampling model to predict the sensitivity of student grades to admitted student standardized test scores.
• The grade susceptibility to standardized test scores is the core element.

​

• We apply the model to fourteen years of undergraduate population data from a large US public university that became increasingly selective over time.
• Selectivity accounts for half of the overall rise in undergraduate grades.

​

• Removing the influence of selectivity reveals academic domain differences in the magnitude of faculty-led grade inflation over the fourteen-year period;
• Low of 0.05 ± 0.01 in business & engineering to high of 0.18±0.01 in the humanities indicates a nearly factor of four dynamic range.

23

Some questions raised...

• What drives domain differences? Deans, Budget model, Faculty attitudes, etc.

​

• Do similar domain differences in faculty-related grade rise exist on your campus? What might the grade susceptibility to pre-college score, Pr(g|s,t), look like?

​

• How might we move Beyond Grades ?

24

Topic 2:

​

Patterns of Gendered Performance Differences in Large Introductory Courses at Five Research Universities

​

Rebecca L. Matz, Benjamin P. Koester, Stefano Fiorini, Galina Grom, Linda Shepard, Charles G. Stangor, Brad Weiner, Timothy A. McKay

​

​

​

​

​

https://doi-org.proxy.lib.umich.edu/10.1177/2332858417743754

26

How do men and women in higher ed compare?

gendered performance differences vs gender performance differences

​

men and women vs male and female

27

​

http://nces.ed.gov/programs/coe/indicator_cha.asp

​

Women far outpace men in terms of enrollment...

28

https://nces.ed.gov/programs/coe/indicator_ctr.asp

…and degree completion within six years

29

http://www.prb.org/Publications/Articles/2011/gender-gap-in-education.aspx

…and degree completion within six years

30

Hill, C., Corbett, C., & St Rose, A. (2010). Why so few? Women in science, technology, engineering, and mathematics. American Association of University Women. 1111 Sixteenth Street NW, Washington, DC 20036.

Yet gaps remain in math, engineering, �computer science, and some physical sciences

31

A Chemical Imbalance / Marie Lidén and Siri Rødnes

Yet gaps remain in math, engineering, �computer science, and some physical sciences

32

How does student performance in large, introductory courses differ by gender across 5 universities?

• 5 universities (A, B, C, D, E)
• 6 years of data; Fall 2008 - Spring 2014
• Developed comparable course grid
• 1,178,517 records from 258 courses
• Student-level data were maintained locally

33

• Each institution handles various populations differently: e.g. honors, majors, living-learning communities
• Some institutions have separate general education courses (“History of Life”) and some are combined
• Students transfer more often between some institutions than others

Developing a comparable course grid �is easier said than done

34

Defining a simple measure

Grade – GPA in other courses = grade anomaly

​

Positive grade anomaly = grade “bonus”

Negative grade anomaly = grade “penalty”

Koester, B. P., Grom, G., & McKay, T. A. (2016). Patterns of gendered performance difference in introductory STEM courses. Retrieved from https://arxiv.org/abs/1608.07565

35

On average some courses yield penalties and others yield bonuses

Grade is usually lower than GPA

→ avg penalty is -.75

Grade is usually higher than GPA

→ avg bonus is .44

R. Sabot, J. Wakeman-Linn, The Journal of Economic Perspectives pp. 159–170 (1991).

Data for 5,830 students that took BS161 between FS11 and SS14 and 12,983 students that took CEM161 between FS08 and SS14.

​

N = 5,830

N = 12,983

Mean grade in course

Mean grade in course

36

What happens when these data are parsed by gender?

Grade is usually lower than GPA

→ avg penalty of .75

​

AGA for Women = -.85 (penalty)

AGA for Men = -.63 (penalty)

​

Average grade anomaly_female� – average grade anomaly_male

= gendered performance difference

​

Gendered performance difference = -.22 (favors men)

Men

Women

Mean grade in course

37

Could these differences be due to �factors besides gender?

Employed regression and matching analyses with these other predictors:

​

• GPA in other courses
• ACT or SAT Math subscore
• ACT or SAT English subscore
• Term

Koester, B. P., Grom, G., & McKay, T. A. (2016). Patterns of gendered performance difference in introductory STEM courses. Retrieved from https://arxiv.org/abs/1608.07565

38

Gathering data from five universities

1,178,517 enrollments across 258 introductory courses in 13 disciplines

COURSES

ENROLLMENTS

39

178 courses representing 714,017 course enrollments

The majority of lecture (74%) and mixed (93%) courses yield a penalty and favor males. The majority of lab (64%) courses yield a bonus.

Grade bonus

Grade penalty

Women favored

Men favored

AGA

GPD

40

Biology

Chemistry

Math & stats

Physics

What is different about math and statistics?

41

Similar GPAs, different grades

Different GPAs, similar grades

Gen chem lecture I at institution D

Cell & molecular lab at institution C

42

Grade bonus

Grade penalty

Women favored

Men favored

AGA

GPD

Accounting / economics

Comm / Pol Sci / Psy / Soc

Writing

43

Gendered performance differences are �small but reliably present

• The lecture/lab pattern is reproducible across biology, chemistry, and physics
• What factors might contribute to these gendered performance differences?

44

Do assessment methods make a difference?

Biology

Bio 1

Mostly MC assessments

Bio 2

Mostly written assessments

45

Grade bonus

Grade penalty

Women favored

Men favored

AGA

GPD

Does collaborative work or class size make a difference?

These engineering design courses in practice might look more like labs than traditional lectures.

46

What lessons have we learned?

• Something is going on in many of our large science lectures
• Running parallel analyses is challenging
• Differences among our universities complicate this work

Topic 3:

Using equity data to promote change in classrooms

• Eric Bell - Arthur F. Thurnau Professor, Department of Astronomy, University of Michigan
• Cait Hayward - Associate Director for Research & Development, Center for Academic Innovation, University of Michigan
• Heather Rypkema - Assistant Director, Foundational Course Initiative, University of Michigan
• Becky Matz - Research Scientist, Center for Academic Innovation, University of Michigan
• Nick Young - Postdoc, Center for Academic Innovation, University of Michigan
• Tim McKay - Arthur F. Thurnau Professor of Physics, Astronomy, Education, and Associate Dean for Undergraduate Education, University of Michigan

48

Course Equity Reports

We are trying to improve equity by letting instructors know about their students, their pathways, and grade equity.

​

Course Equity Reports:

• 16-page report focused on a course
• Historical data depicts student experiences and outcomes
• Reveal outcomes as associated with barriers
• Prompt reflection and provides resources and ideas for how to address inequities

Equity is not a single individual’s effort, but is situated within a program and a domain and a school.

Outcome disparities: Course

Outcome disparities: U-M

Barrier Index:

Crude, ‘intersectional’ sum of factors that are institutional barriers to success ( First Generation, Low-income, Under-represented Minorities)

50

Equity reports are used in both intensive and broader efforts

Foundational Course Initiative

• 3-year course reform focused on few large courses
• Nearly 30% of undergrads in >=1 FCI course

Pushed to instructors

• Urgency to inform instructors with move to remote learning
• Pushed to instructors of large and inequitable classes
• Still very much a work in progress

Current Research

Faculty Focus Groups

• How do faculty interpret the reports in their current form?
• How do faculty change their curriculum to address equity issues identified?

​

Cross-course learning analytics research

• What curricular structures foster equitable assessment practices and outcomes?

​

​

51

Thank you! Questions?

​

52

Gus Evrard - evrard@umich.edu

​

Cait Hayward - cholma@umich.edu