Research Partnerships @ Code.org

Baker Franke

SIGCSE 2021

​

Goal: Invest in supporting high-quality computer science education research that leverages Code.org’s data and tools.

​

Supporting Advancement in CS Ed Research

Supporting Regional Partners

Supporting our Mission

Education Programs @ Code.org

Course Activity Data

User Demographic Data

PD Activity Data

Survey Data

2(ish) models of partnership

• Data sharing
• “Easy” - we have data - want some?
• “Hard” - you want all student code?
• Lack Capacity
• Privacy concerns
• High demand for K5 puzzle data

​

2(ish) models of partnership

2. Implementation -- we implement a researcher’s intervention on our platform and give them data back. (Collaborative)

• “Easy” - e.g. add a few survey questions.
• “Hard” - implement a 2x2 RCT requiring building of multiple levels and/or an entire alternate version of a unit with a new tool palette

Why you want to partner with us - Infrastructure and Scale

• Linking out
• E.g. passing user_id to an app
• Research/Cohort group “tracking”
• Data for all students under PD’d teachers with Regional Partner X
• Randomized experimental design (A/B testing)
• Student A sees one quiz question, Student B sees the other...at the “teacher level”
• Arbitrary experiments data
• E.g. number of times hit block/text toggle

Partners and Results

Data Sharing -- Chris Piech, Stanford University

We are able to provide autonomous feedback for the first students working on an introductory programming assignment with accuracy that substantially outperforms data-hungry algorithms and approaches human level fidelity. Rubric sampling requires minimal teacher effort, can associate feedback with specific parts of a student’s solution and can articulate a student’s misconceptions in the language of the instructor.

Swipe Right for CS: Measuring Teacher Bias about Recruitment into Computer Science

Joshua Littenberg-Tobias, Kevin Robinson, Gabrielle Ballard

Teaching Systems Lab, MIT

April 2018

What is Swipe Right for CS?

• Users are presented with a profile and a series of arguments to persuade students to take computer science
• Users “swipe right” if they think the argument would be effective in convincing the student and “swipe left” if they do not find it convincing.

You can see these differences in how likely users were to swipe right for specific groups of students compared to white students.

Implementation - David Weintrop Univ. Maryland

Do students answer these differently?

​

Does the programming environment matter?

​

​

Implementation - David Weintrop Univ. Maryland

​

. Our analysis shows students performing better on questions presented in the block-based form compared to text-based questions. Further analysis shows that this difference is consistent across conceptual categories.

Subgoal Labels Study -- What are subgoals?

Research Partner: Briana Morrison - U. Nebraska - Omaha

Subgoals:

• Program comments that describe steps in a problem solving process
• Students asked to break problem down, and compose solution with “subgoals” before writing code
• Helps thinking about writing code - gets novice “over the hump” - bridges the gap between expert and novice (i.e. the gap between a teacher’s explanation and student’s understanding)
• Research has shown that subgoal labels a) improve student performance on programming tasks b) especially for women and URMs

Subgoal Labels Study -- How it worked

Participants opt-in (randomly selected: intervention + control)

• Participating teachers assign a research version of Unit 3 to students through the course pull down, and teach as normal.
• The URL will look something like: studio.code.org/s/csp3-research-XQEQYV

We will share data with the researcher (deidentified) for students of teachers in the study. There will be no way for researcher to know the identity or any PII of any teachers or students.