Teaching Creative and Practical Data Science at Scale
Shannon E. Ellis
Associate Teaching Professor
UC San Diego
Cognitive Science
Halıcıoğlu Data Science Institute
IMSI Teaching and Evaluating Data Communication At Scale
2024
Our approach to and lessons learned from 5,000+ students in COGS 108
@Shannon_E_Ellis
sellis@ucsd.edu
shanellis.com
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
COGS 108
Today’s Plan
My (not-so-secret) goal: Share what we’ve done to learn from all of you...so please do share your thoughts/ideas on what I share here.
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
COGS 108
COGS 108: Data Science in Practice
COURSE OBJECTIVES
Formulate a plan for and complete a data science project from start (question) to finish (communication)
Explain and carry out descriptive, exploratory, inferential, and predictive analyses in Python
Communicate results concisely and effectively in reports and presentations
Identify and explain how to approach an unfamiliar data science task
more simply: THINK about the thing, DO the thing, and be able to COMMUNICATE about the thing
Donoghue T, Voytek B, and Ellis SE. Course Materials for Data Science in Practice. The Journal of Open Source Education. (2022)
Upper-division, large-enrollment (~450/section) undergraduate course
Single prerequisite: intro programming course
Focuses on application (less on theory)
Quarter system: 10 weeks + finals
Course staff: 1 Instructor, 3-4 TAs, 3-4 IAs
1-2 sections offered every term since 2017
COURSE OBJECTIVES
Formulate a plan for and complete a data science project from start (question) to finish (communication)
Explain and carry out descriptive, exploratory, inferential, and predictive analyses in Python
Communicate results concisely and effectively in reports and presentations
Identify and explain how to approach an unfamiliar data science task
more simply: THINK about the thing, DO the thing, and be able to COMMUNICATE about the thing
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
COGS 108
COGS 108: Guiding Principles
Course Goals Described in Detail: Donoghue T, Voytek B, and Ellis SE (2020). Teaching Creative and Practical Data Science at Scale. Journal of Statistics and Data Science Education
Our students are really good at completing assignments.
They’re comfortable when there’s a right answer and they get points for said right answer.
They struggle with open-ended analyses.
They find discomfort when there’s not one “right” approach.
They struggle (initially) to go from a general thought/idea to a specific, data-driven question.
And, they struggle (initially) to consider ethical implications beyond data privacy.
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
COGS 108
COGS 108: Course Components
Lecture (~3h/wk) - conceptual + technical instruction; clicker questions + lecture quizzes
Discussion (1h/wk) - technical instruction; weekly lab completion
4-6 technical individual assignments (python, pandas, linear regression, text analysis, machine learning, etc)
Quarter-long Group Project
COURSE TOPICS
...and 2 guest lectures
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
COGS 108
COGS 108: STUDENTS
PROGRAMMING BACKGROUND
Upper-division: 95% 3rd and 4th years
Area of Study: 32 different majors; top 3: 1) Cognitive Science (40%), 2) Computer Science (30%), 3) Data Science (15%)
Gender: 40% female; 2% nonbinary
Race: 78.6% Asian; 20% White; 1% Black; 1% American Indian/Alaskan Native; 1% Native Hawaiian/Other Pacific Islander (11% Hispanic/Latino)
International Students: 25%
Transfer Students: 20.2%
First-gen college student: 25%
I've never programmed
I could teach an intro programming course
These are data from Fall 2023, but these patterns hold across terms.
COGS 108
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
STUDENTS: Prior Knowledge
COURSE OBJECTIVES
STATS & TOPIC FAMILIARITY
(PRE)
(POST)
Disclaimer: NOT filtered to only include the same students pre- and post
What's statistics?
I could teach an intro stats course.
COGS 108
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
STUDENTS
COGS 108: DETAILS
QUIZZES, LABS & ASSIGNMENTS
Quizzes (9) - multiple-choice, timed, open-notes/open-Internet; formative assessment on lecture content
Labs (8) - open-ended; build upon previous week’s content; graded on concerted effort (errors are fine; incomplete is fine; looking for ~1h of dedicated effort; answer keys provided)
Assignments (4) - prescriptive; summative assessment; notebooks guiding students through an analysis; autograded using nbgrader
TERM-LONG PROJECT
~8 weeks; groups of 4-5 students carry out the entire data science process on a topic of their choosing
Components: Previous Project Review*, Proposal, Data Checkpoint*, EDA Checkpoint*, Final Report, Final Video*, Feedback Survey* (+optional weekly surveys)
Each Group Provided: Instructions, template files, access to a private GitHub repository
*Not required every course iteration
COGS 108
STUDENTS
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
ACTIVE LEARNING: Data Intuition
THE DATA
In a lecture on Data Intuition, students complete a short form at the end of class...(end of wk2)
BRAINSTORM, CLEAN, ASSESS
Activity Credit: Brad Voytek
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
COGS 108
STUDENTS
ACTIVITY: THINK -> DO
DO: LAB INSTRUCTIONS
THINK: In class, propose a very general question “Does Congress Have an age problem” and talk/think through the entire process, taking notes as we go.
DO: STUDENTS COMPLETE
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
COGS 108
STUDENTS
PROJECT: Final Report
Students work to complete a full data science project from question formulation to written (report) and oral (presentation) communication.
IMPROVE
LESSONS
EVALUATE
DO & SAY
DETAILS
COGS 108
STUDENTS
DO:Project Examples
The Pandemic Playoffs: The Effects of the COVID-19 Bubble on NBA Playoff Game Statistics (Sp21)
Offensively, the bubble eliminated home-court advantage; Defense improved in the bubble
COVID-19 Impact on UCSD Students (Wi21)
Students report higher class recommendation rates and receive higher grades during COVID-19
Home Teams
Away Teams
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
DO: Projects Summarized
Typical notebook contains 92 cells, 320 lines of code, 3,175 words of text, 8 external imports, and 6 external links
Gollamudi and Ellis (in prep)
Almost every project imports pandas, matplotlib, and numpy
Markdown to start, 50:50 at 20-80%, Markdown to finish
Student notebooks have 3.7x more cells, 3.8x more lines of code, and 14.6x more words of text than the typical publicly-available notebook (Rule et al., CHI 2018)
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
DO: Project Topics
Most subtopics remain relatively constant, with a single subtopic typically making up no more than 15% of total projects in a given quarter.
Main project topics remain relatively consistent across quarters, with entertainment, communities, and public health making up the majority of projects.
Gollamudi and Ellis (in prep)
22%
15%
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
DO: GitHub History
When checkpoints were required, students typically agree they were helpful.
Anecdotally, I got wayyyyy fewer panicked messages at the end of the term when checkpoints were implemented.
Consistent pattern across course iterations.
Suggests that 1) effort is focused around project deadlines and/or 2) students push changes to GitHub for deadlines.
Gollamudi and Ellis (in prep)
Project Proposal
Data Checkpoint
EDA Checkpoint
Final Report
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
SAY: Student Survey Responses
Students self-report spending ~22 hours throughout the term on the project
Student responses to: “Approximately how many hours did you personally spend working on your final project in total this entire quarter (this includes brainstorming time, working on proposal, group meetings, time spent on own, writing report, etc.)?"
Student responses to: “Respond to the following about YOUR feelings toward the COGS 108 Final Project.”
Gollamudi and Ellis (in prep)
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
SAY: Stress and Checkpoints
No obvious difference in students’ reported stress level when checkpoints were and were not included.
Student responses to: "Working with my team was more stressful than I anticipated."
Reported Notebook Metrics Between Remote Quarters With and Without Required Checkpoints
No clear difference in student notebook metrics when checkpoints were and were not included.
STUDENT COMMENT:
Thought the checkpoints were great for helping us get our work done. Wish there could be a Part II to this class where we could continue to build off our projects with our teams, but I understand that'd be hard to coordinate. Appreciate both the freedom we recieved [sic] in our topic and the structure of the checkpoints!"
Gollamudi and Ellis (in prep)
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
SAY: Student Comments
All in all, the project was a bit of fun for my group. We were able to find a topic we all liked, on a personal level, and were able to draw some relatively interesting conclusions based on our datasets/statistical analysis. The best part of the project was definitely the ability to select our own topic, which really motivated the team to work collaboratively.
As someone that typically dislikes group work I really enjoyed the project as well as working with my group. Coming to the end of the journey is somewhat of a bittersweet moment, but I genuinely feel like I learned a lot along the way.
I enjoyed the ability to choose the topic and group when working on the project. I felt that this helped motivate me to work on it more.
It was hard for me to find interesting data. Especially, it was hard to find local, accurate, highly robust data.
I feel like a lot of people here were inexperienced with working with Github and that was a contributor to why people had trouble getting their parts done. Our overall division of work could have been more fine tuned so that everyone had a better idea of what exact tasks people were expected to do.
I like having this team evaluation portion because I have a bad experience with group project when some of the members avoid contributing to the project. I think having this portion encourages everyone to actually contribute to the project.
Although I enjoyed and learned a lot from completing the assignments, I felt that the final project was beyond the scope of my capabilities. I don't think it would have been possible for me (and I imagine, many others who aren't experienced in data science/statistics) to have performed confidently well on the final project without the help of others.
COGS 108
STUDENTS
DETAILS
IMPROVE
LESSONS
EVALUATE
DO & SAY
EVALUATE: Projects
PROJECT GRADING: OUR FOCUS
Single grader throughout the term (proposal, checkpoints, report)
GRADING TIMELINE
Week | What |
5 | Review |
6 | Proposal |
8 | Checkpoint #1 |
9 | Checkpoint #2 |
11 (Finals) | Report & Video |
TOOLING
DETAILED HUMAN-GENERATED FEEDBACK when most important [ Projects ]
AUTOGRADE [ Assignments ] or SPOT-CHECK [ Labs ] everything else
COGS 108
STUDENTS
DETAILS
DO & SAY
IMPROVE
LESSONS
EVALUATE
FEEDBACK: Example
FINAL REPORT
Overview, Question, Background & Hypothesis | background OK, but a lot more has been done/studied on this topic, so this is not fully complete as is;
Data Description | well done
Data Cleaning/Processing | comments are best for describing code; a bit more markdown text to summarize would help guide the viewer as to the process; need more organization here generally to guide the viewer - the work done is good, but the report aspect is unclear
Data Visualization | axis labels are quite small; throughout; "(probably Los Angeles)" <- why not verify?; do the colors on the housing price vs. homeless count help us understand which point is which country? This color scale would only help if the colors mapped to a variable (i.e. population); unsure what "list outliers" are?; your population boxplot looks as if population hasn't varied at all? is this correct?; I don't love the removal of LA generally (since those data are real/correct/exist) and it's unclear from explanation if it's removed just for that plot or overall.
Data Analysis & Results | watch typos throughout ('of homeless'); what question is the model answering? not clear; unclear why outliers are dropped; models not interpreted...or explained? why are there so few points on your plot? what about time?
Ethics & Privacy | "We must cross reference statistical information within our chosen dataset with outside sources to verify its validity" <- Did you do this? "certain high density homeless areas may not be able to collect accurate counts" <- Did you find this? Seems like this section wasn't updated after the proposal.; would have loved to see a regional analysis and other variables looked at in your analysis
Conclusion & Discussion | I think you're confusing p-value and r (correlation) when you say "P-value of 0.037. Which by no means is a sign of significant correlation"
QUESTION
Is an increase of a city's housing prices in California correlated to rates of homelessness in that city?
PROPOSAL
Question | I really like this question overall and think it's quite important. As you zero in be sure to include where you're asking this question...in CA? across the US? Somewhere else?
Background | This is a good start, but if focuses more on the housing aspect...what about the unhoused population? What do we need to know about that and trends of either over time? Flesh this out a bit more as you go this quarter to really fully introduce your topic and both sides of the question.
Hypothesis | ok (going to add in an additional thought here: what about weather/location/etc? Do you think this effect will be the same nationwide? Are there other factors to consider?)
Data | If it's your ideal dataset, you get to specify how many years you want! And, it's unclear what each row would be...three different datasets for state, country, and city? All together? If so, how? Anything else you want to know about the specific city/state/country? A good start here, but needs a bit more.
Data Ethics/Privacy | well done
Expectations & Timeline | good
Overall | clear and well written
In the proposal, students often struggle:
In the final report, common high level struggles include:
COGS 108
STUDENTS
DETAILS
DO & SAY
IMPROVE
LESSONS
EVALUATE
EVALUATE: Assignments
AUTOGRADE PROGRAMMING ASSIGNMENTS (nbgrader)
STUDENT FACING
c
STUDENT FEEDBACK
(Note: different assignment)
In a single document: solutions & tests
Enables: visible and hidden tests, partial credit (points assigned per test cell)
COGS 108
STUDENTS
DETAILS
DO & SAY
IMPROVE
LESSONS
EVALUATE
COGS 108: LESSONS
COURSE PLANNING:
SPEND TIME PLANNING FOR FIRST ITERATION.
TECH STACK AS SIMPLE BUT MODERN AS POSSIBLE.
AUTOGRADE WHAT YOU CAN. PRIORITIZE DETAILED FEEDBACK WHERE MOST IMPACTFUL.
SMALL CHANGES HAVE LARGE DOWNSTREAM CONSEQUENCES.
PROJECT SPECIFIC:
PROVIDE STRUCTURE
MOTIVATE THEIR INTERESTS
PROVIDE SPECIFIC FEEDBACK
REMIND STUDENTS THAT: STRUGGLING IS TYPICAL, WRITING CODE IS NOT THE ONLY THING THAT COUNTS, AND TASKS WILL TAKE LONGER THAN EXPECTED
TRUST MOST WILL PARTICIPATE
COLLECT & USE STUDENT FEEDBACK
YOUR THOUGHTS, FEEDBACK, ADVICE:
COGS 108
STUDENTS
DETAILS
DO & SAY
EVALUATE
IMPROVE
LESSONS
IMPROVE & LEARN
GROUP FORMATION: self-selection, instructor formed, random, pairs-> groups
The literature suggests heterogeneous groups without “isolated” students.
We do not know what’s best.
We have started to study this.
Fa23 & Wi24 (currently): studying gender breakdown of group on student feedback and project output
Current findings: students really hate being put into groups
Data analysis is underway.
VALUE & IMPACT OF FEEDBACK: Do students who receive “better” feedback improve more than those who receive worse? What types of feedback are most helpful? Does this differ by student/group?
We give a lot of students a lot of feedback.
Quality, amount, length, depth, etc. differ among graders.
We can study this.
...but we’re currently focusing on group formation.
COGS 108
STUDENTS
DETAILS
DO & SAY
EVALUATE
LESSONS
IMPROVE
IMPROVE & LEARN
(SOME OF) WHAT WE CAN DO BETTER
Version control | Teach, scaffold and support git usage better
Communication | Provide opportunity for presenting live; prioritize communication more throughout
Motivate (& require) iteration | Have students reflect on and respond to feedback (Yes, Ramona)
Code Products | Move toward clean and documented code...rather than just “it works” (Thanks, Allison!)
Update material more quickly as material gets stale and/or less relevant
LLMs | Incorporate more direct instruction and demonstration on usefulness (and cautions) of LLMs/AI for data science
Teach Smaller Classes | hah! (but actually...it’s sometimes possible - listen to Suraj tomorrow!)
Improve student lecture participation | varies widely across course iterations
Actually check for academic integrity/plagiarism violations | we’re just not doing this in any systematic way...
Teach less, demo more | inspired by Mine Çetinkaya-Rundel
COGS 108
STUDENTS
DETAILS
DO & SAY
EVALUATE
LESSONS
IMPROVE
QUESTIONS?
ACKNOWLEDGEMENTS
Bradley Voytek & Jason Fleischer - Instructors
Sashwath Gollamudi | Project Analysis
Scott Yang | COGS 108 Tooling Dev
COGS 108 students! All 5,024 of you (to date)!
Former COGS 108 Instructional Staff Members - TAs and IAs
Data Science educators who make their work and share their knowledge openly!
National Science Foundation (Grant No. NRT-IGE-1735234)
@Shannon_E_Ellis
shanellis.com
RESOURCES:
COGS 108 GitHub: https://github.com/COGS108
Projects: https://github.com/COGS108/Projects
Tom Donohue, Bradley Voytek and Shannon Ellis (2021). Teaching Creative and Practical Data Science at Scale. Journal of Statistics and Data Science Education. [link]
Adam Rule, Aurélien Tabard, Jim Hollan (2018). Exploration and Explanation in Computational Notebooks. CHI. [link]
Thoughts/feedback/advice: http://bit.ly/imsi_ellis
COGS 108
STUDENTS
DETAILS
DO & SAY
EVALUATE
LESSONS
IMPROVE
Template created by: SlidesMania.Com