Frequent Small Assessment

All the Reasons

— Geoffrey Challen (on behalf of CS 124)

Only Frequent Small Assessment

All Some of the Reasons

— Geoffrey Challen (on behalf of CS 124)

White

Male

Cisgender

Heterosexual

Well-Paid

Able

INTJ

Ultimate frisbee player

…etc

​

challen@illinois.edu

geoffreychallen.com

learncs.online

0.0. Terminology

• Frequent Small Assessment
• High-Stakes Assessment
• Only Frequent Small Assessment
• => No high-stakes assessments
• => No midterm, no final

0.1. Visualizing Frequent Small

0.1. Visualizing Frequent Small

Midterm (40%)

Final (60%)

Traditional Model

0.1. Visualizing Frequent Small

Midterm (40%)

Final (60%)

Traditional Model

0.1. Visualizing Frequent Small

Midterm (40%)

Final (60%)

0.1. Visualizing Frequent Small

CS 124

(Only Frequent Small)

~80 daily exercises (0.4% each)

15 quizzes (2.6% each)

1 project (7 weekly checkpoints, 4.2% each)

0.1. Visualizing Frequent Small

CS 124

(Only Frequent Small)

Student Preparation

0.2. Problems with High-Stakes Assessments

• Cramming
• Element of surprise
• Exam administration
• Student stress and anxiety
• Steals points from frequent small assessments

0.3. Does It Work?

Data from CS 124 Fall 2020–Spring 2022 (asynchronous online format)

• Total enrollment: 3,344
• 50% more than previous 2-year period
• 80% non-majors (50% acceptance rate)
• Pass rate: 96.5%, 79% A-range grades
• Experience performance gap: 1% typical, 0% observed
• Defined as between students with no prior experience and with any prior experience
• Strong continuation and performance in downstream courses for both majors and non-majors
• Inherited a course that was already headed in this direction
• Having 15 computerized proctored quizzes to write questions for is a nice problem to have…

All the Reasons for Frequent Small Assessment

​

1. Students do better
2. Preparation drives learning
3. Supports successful strategies
4. Help students not get behind
5. Support students proactively
6. Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

Enter a number between 1 and 12:

|

1. Students Do Better

​

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Stick around for the talk at 2PM by Craig's research group
• Measured improvement in performance achieved by FSA compared to HSA
• But this probably understates the case
• HSA assessment results artificial boosted by cramming
• Lots of other benefits! (#2–12)
• Short-term memory is the ultimate confounder with HSAs

2. Preparation Drives Learning

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Students prepare for assessments
• Preparation drives learning
• Spaced preparation ~ spaced repetition, which has been shown to be highly effective
• Why don't we talk about this more?

3. Supports Successful Strategies

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Ask yourself: How would you tell a student to approach succeeding in your course?
• Design your grade structures accordingly
• Press # for mini rant on self-regulation
• Midterm + Final isn't like "the real world"
• Faculty are poorly prepared to make claims about self-regulation

4. Help Students Not Get Behind

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Assessment frequency limits the degree to which students can unknowingly fall behind in your course
• 8 weeks in: way too late
• 2 weeks in: salvageable
• Make sure students know what they don't know as soon as possible so that they can learn what they need to know
• Side note—
• Forces instructors to deal with pacing issues
• If only 50% of the class meets a deadline, you have a problem with your deadline

5. Support Students Proactively

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Use data to spot and approach students who are struggling early
• Replicate the feeling of a small class
• Can help; doesn't always
• But make sure students who are having a hard time know that they are seen
• Great for introductory courses, for first-year students, and for students who have trouble transitioning to college
• Everyone, really

6. Improves Course Design

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Data from tightly-scoped assessments points directly at problems with course sequencing and pacing
• (Story from CS 124)

7. Student-Friendly Grading Policies

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• More assessment data allows for student-friendly grading policies
• Dropping low scores (discard outliers)
• Second-chance assessment (oversampling)
• Catch-up grading (smoothing)

8. Students Learn How to Learn

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• More assessments gives students more chances to learn how to study and prepare
• Particularly when combined with student-friendly grading policies like dropping low scores
• Also allows students to become comfortable with proctoring environments and processes
• HSAs don't provide enough practice or reliable results

9. Support Student Preparation

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Unlearn the HSA mindset
• Telling students what is on the quiz is OK
• Telling students what is on the quiz will help them prepare
• Telling students what is on the quiz will reduce anxiety
• Telling students what is on the quiz is good!
• CS 124
• Every quiz has a practice quiz
• The practice quiz is very similar to the actual quiz

10. * Probably Less Stressful

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Two high-stress events v. 15 low-stress events
• Hypothesis: stress is not additive
• FSA becomes routinized, HSA is always unusual
• Now incorporate the effects of
• student-friendly grading
• availability of preparation materials
• no secrecy about what is on the exam
• directly incentivized preparation

11. * Course Feels Easier

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• What are you more likely to remember?
• Running 1 mile a day for 30 days
• Running 30 miles in one day
• By scaffolding student effort students achieve more while feeling like they are doing less
• Can use this as an opportunity to cover more material
• Or just let the course feel easier
• CS 124 students write way more code than they used to, with similar perceptions of difficulty

12. Simplifies Course Administration

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

• Absences can be easily accommodated by drops
• No more doctor's notes
• No more death certificates
• Enables a much more empathetic way of working with students
• Frequent doesn't mean rigid
• Can build in flexibility to allow students to create gaps for other things without getting too far behind

All the Reasons for Frequent Small Assessment

​

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration

Enter a number between 1 and 12:

Control-C

2.0. What You'll Need

• Lots (and lots) of questions
• Breadth before depth (coverage before variants)
• No need to use any specific tool or framework
• Examples from CS 124, per semester
• ~70 homework programming exercises (homework)
• ~45 quiz "" (quizzes)
• ~45 practice quiz "" (practice quizzes)
• ~++ for extra practice
• Currently have ~620 programming problems
• Big investment in technology support in this area to accelerate problem development
• Also novel debugging challenges, and testing problems coming soon
• Opportunities for inter-institution collaboration

2.1. What Would Help

• Institutional acceptance
• Time to start de-embedding the "traditional" model
• Institutional recognition
• Focus on grade design in teaching evaluation
• (Deemphasize "the show")
• Institutional support
• Help with logistics and exam management
• Particularly for large courses
• FSA-friendly policies
• Testing centers
• But a cautionary note…

2.2. All That Said

You can move to FSA-only today using your existing course schedule, facilities, and infrastructure

2.3. CS Can Lead

Computer science is uniquely positioned to lead on FSA

​

• Autograding
• Computerized testing
• Skill-based courses
• Lots of student interest
• Right skill set

All the Reasons for Frequent Small Assessment

​

• Students do better
• Preparation drives learning
• Supports successful strategies
• Help students not get behind
• Support students proactively
• Improves course design

• Student-friendly grading policies
• Students learn how to learn
• Support student preparation
• * Probably less stressful
• * Course feels easier
• Simplifies course administration
• Questions?

Enter a number between 1 and 12:

13

challen@illinois.edu

geoffreychallen.com

learncs.online

go.cs.illinois.edu/teaching-faculty-hiring