Motivation

01

Limited Generalizability

Prior work evaluates predictive power of programming log data features within a single course or environment.

​

Motivation

01

Limited Generalizability

Prior work evaluates predictive power of programming log data features within a single course or environment.

​

Motivation

02

Standardization Obstacle

Features are defined and computed differently across papers, makes it difficult to replicate or compare results.

​

Hoq et al. (2023)

Average score on students' assignments correlates with final course scores.

Leinonen et al. (2017)

Active Time correlates with assignment scores, but correlates less with exam scores.

Watson et al. (2014)

Watwin score and the percentage of lab time spent on resolving errors correlate with course performance.

kerschbaumer et al. (2023)

Number of comments and source code complexity has a positive influence on students' success.

Paper Goals

01

Evaluate the predictive validity of commonly used features across different course contexts.

02

Explore whether simple features are more or less powerful than ones that

require additional logging or analysis.

RQ1

Which features derived from students' programming log data early in the course are most predictive of students' course outcomes?

How consistently do these features perform across datasets?

Attempts

The number of times the student submitted their code for feedback on a given problem

MaxScore

The highest score (0-1) achieved across all submissions for a given problem

FirstCorrect

Binary indicator if the first submission for a problem was successful.

EverCorrect

The proportion of all \textit{attempted} problems that the student ever completed correctly

ProblemsStarted

The number of problems that a student started.

StartTime

The time of the first log entry for the student's work on a given problem.

FirstCorrectTime

The time of the first fully correct submission (i.e. passing all test cases) for a given problem.

ActiveTime

The time the student spent actively working on this problem, ignoring gaps that may indicate disengagement or breaks.

PassiveTime

The amount of time the student spent in shorter gaps in programming activity (with no log data) on this problem.

TotalTime

The sum of ActiveTime and PassiveTime.

CWO (Java) - US Public R1

Small exercises, any-time submission. Includes student final exam scores.

Falcon (Python) - US Air Force Academy

Logs testing and grading submissions. Exam scores calculated from problem-labeled questions.

Edwards (Python) - Mid-sized US

Major/non-major students. Analyzed final graded submissions only. Includes final course grades.

CB (Python) - UFAM Brazil

Online Judge (CodeBench) logs keystrokes and submissions. Includes exam coding scores.

1. Early Prediction Window

• If due dates available: First 30% of assignments used; and the cutoff is the last due date in this set.
• Otherwise: Activity distribution analysis identifies the most prominent gap (20% - 40% window).

2. Data Aggregation & Normalization

• Features calculated per student per problem.
• Z-score normalization applied within each problem.
• Then we averaged z-scores across all problems for a single student feature value.
• This choice is because z-scores have a similar scale for every problem, while raw values may have different scales on each problem

3. Correlation Analysis

• We used Spearman correlation to measure the relationship between normalized student feature values and final course grades.

Results - Correlation Analysis

01

Student Scores on Assignments

Early assignment scores represent the most reliable predictive feature for student outcomes across all datasets analyzed.

02

Early Start Time (Procrastination)

Students who start problems earlier (lower StartTime) consistently perform better in the course, aligning with prior research on the negative impact of procrastination on success.

01

Little consistency across datasets

Max Score and Start Time are the most reliable across datasets.

02

Some consistency between datasets

• Passive Time is consistent across Falcon datasets, but not in CWO and Edward's datasets.
• Active Time consistent across CWO and Falcon datasets, but not in Edward's.

• What data is included? The impact of short exercises is different from the impact of scores of projects on final course outcomes.

• What data is included? Variations in log granularity and logging triggers across different platforms.
• What outcome measure is used? Differences in how "student success" is defined (e.g., final exam vs. course grade).

• What data is included? Variations in log granularity and logging triggers across different platforms.
• What outcome measure is used? Differences in how "student success" is defined (e.g., final exam vs. course grade).
• Are there any additional interventions? The presence of automated feedback or human tutoring impacts student programming behaviors.

01

The importance of evaluating predictive analytics on multiple datasets highlights features that can be broadly adoptable, or should be generalized with caution.

01

The importance of evaluating predictive analytics on multiple datasets highlights features that can be broadly adoptable, or should be generalized with caution.

02

Data sharing and documentation are essential to enable secondary analysis and support understanding of the results.

01

The importance of evaluating predictive analytics on multiple datasets highlights features that can be broadly adoptable, or should be generalized with caution.

02

Data sharing and documentation are essential to enable secondary analysis and support understanding of the results.

03

Standardization of log data features calculations to facilitate replication and secondary data analysis

01

The importance of evaluating predictive analytics on multiple datasets highlights features that can be broadly adoptable, or should be generalized with caution.

02

Data sharing and documentation are essential to enable secondary analysis and support understanding of the results.

03

Standardization of log data features calculations to facilitate replication and secondary data analysis

04

Exploration of how contextual variations impact the predictive power of programming log features.

Samiha Marwan - Postdoctoral Researcher

samarwan@ncsu.edu

Thomas W. Price - Associate Professor in CS

twprice@ncsu.edu