1 of 19

Grouping and Differentiating Programming Exercises Using Minimal Programming Constructs

Ka Weng Pan, Bryn Jeffries and Irena Koprinska

School of Computer Science, University of Sydney

Sydney, Australia

{kaweng.pan, bryn.jeffries, irena.koprinska}

@sydney.edu.au

2 of 19

Context: introductory computer programming course

Learning computer programming is difficult
It requires regular practice, timely feedback and effective sequence of learning activities when designing the course
Introducing concepts in appropriate order, prerequisite concepts before new concepts, gradually increased difficulty

-> to minimize cognitive load, improve success rate

Key contribution: Group programming exercises based on a set of minimal programming constructs (automatically extracted)
Overall goal: Help educators to: design appropriate sequence of programming exercises, identify large knowledge gaps between consecutive exercises

Are the students successfully learning and applying the concepts taught?
Is the sequence of exercises appropriate? What are the key points to intervene?

3 of 19

Method

For each successful student submission (correct solution of an exercise):
Step1. Identify the primitive programming constructs
Step 2. Aggregate the primitive constructs to form a set of minimal primitive constructs representing the set of concepts applicable to solve the problem
Step 3. Using this set, define distance between exercises

This enables:

Automatic identification of the actual set of concepts applied by students to each exercise
Reassessment of exercise sequence – identifying large gaps between exercises, in terms of concepts used

4 of 19

Example

Task (exercise, problem): Write a program to output “123”
Step 1: Identify the primitive constructs used in successful student submissions

Successful student submissions	Primitive constructs for each submission
print("1" + "2" + "3")	[imported function call] print, [binary operation] add, function call, [variable] load, [constant] str
print("123")	[imported function call] print, [variable] load, [constant] str, function call
print(f"{123}")	[imported function call] print, f-string, function call, [constant] int, [variable] load

5 of 19

Example (2)

Step 2: Aggregate primitive constructs -> set of minimal primitive constructs

Successful student submissions	Primitive constructs for each submission
print("1" + "2" + "3")	[imported function call] print, [binary operation] add, function call, [variable] load, [constant] str
print("123")	[imported function call] print, [variable] load, [constant] str, function call
print(f"{123}")	[imported function call] print, f-string, function call, [constant] int, [variable] load

Set of minimal primitive constructs for exercise

[constant] int,

[constant] str,

[imported function call] print,

[variable] load,

f-string,

function call

f-string,

function call

6 of 19

Step 1: Identify primitive construct set

For each successful (correct) submission for an exercise:
1) Generate the abstract syntax tree (AST), 2) Extract node classes from AST, 3) Refine to capture additional features, remove redundant nodes

try:

a = []

print(a[0])

except IndexError:

print("IndexError")

Successful student submission

1) AST

ast.Assign,

ast.Call,

ast.Constant,

ast.ExceptHandler,

ast.Expr,

ast.List,

ast.Load,

ast.Module,

ast.Name,

ast.Store,

ast.Subscript,

ast.Try

2) AST node classes

constant] int,

[constant] str,

[excepthandler] IndexError,

[imported function call] print,

[variable] load,

[variable] store to,

assignment,

function call,

list,

subscript,

try

3) Final primitive set

7 of 19

Step 2: Identify set of minimal primitive constructs

8 of 19

Step 3: Compute distance between exercise

9 of 19

Case study

Beginners

Year 7-8, fundamental computational thinking
40 exercises, attempted by 6,539 students

Intermediate

Year 9-10, data structures, files and functions
25 exercises, attempted by 3,288 students

Exercises are automarked against test cases

Number of passed and failed submissions for each exercise:

Two online programming courses for high school students in Australia

10 of 19

Extracted sets of minimal constructs

Beginners: 102 primitive constructs in total; for exercises1-6:

Intermediate: 154 primitive constructs in total; for exercises 1-4:

11 of 19

Most common constructs in minimal construct sets

Beginners – only the top 2 concepts appear in all exercises; introduces concepts in a more sequential and distinct way
Intermediate – top 7 concepts appear in all exercises; uses a larger common set of concepts – students need to combine these concepts

12 of 19

Per-exercise minimal primitive construct sets�

Showed great diversity of student approaches in some cases
For Intermediate, minimal primitive construct sets were surprisingly large, when compared to intended solutions:
Exercise 1: 28 constructs identified:

Reference solution: much simpler - 11 constructs needed (in bold)

Such big deviations should be investigated -> may need to improve course design and provide better guidance/resources?

13 of 19

Visualisation using primitive construct map

Shows similarities between exercises
Binary: concept present or not
Can be used to identify large increments of new constructs

Beginners, Exercise 1-6

14 of 19

Clusters based on minimal primitive constructs

Clustering based on the same set of minimal primitive constructs
Most of the clusters were trivial – 1 exercise only
3 interesting clusters for Beginners - correctly identified pairs of exercises where the first exercise intended to provide a scaffolding approach, to be used in the second exercise

Clustering can be improved – pruning less frequent constructs or using a tolerance threshold of shared constructs, or by a machine learning approach

Clusters of exercises

15 of 19

Exercise distance vs exercise failure rate

Hypothesis: If an exercise has a big number of new programming constructs, it will have a high failure rate, due to the high cognitive load of applying these constructs
Failure rate: the proportion of syntactically correct submission that failed the tests (were semantically incorrect)
Investigate cumulative distance and consecutive distance vs the failure rate

No strong correlation observed – many factors influence the difficulty of an exercise

Beginners: r_cum=0.06, r_cons=0.31
Intermediate: r_cum=0.26, r_cons=0.07

16 of 19

Interesting cases - 1

Exercises 14 and 22: Large increase in new constructs but decrease in failure rate
Reason: Large number of new constructs were indeed introduced but they are easy to understand – most are from the turtle graphics library

17 of 19

Interesting cases - 2

Exercise 29: Increase in new constructs and increase in failure rate

This exercise requires a mixture of prior constructs – challenging for students with surface understanding of prior constructs
Untaught constructs (max,sort) observed in the set of new constructs – students struggle to determine the constructs being assessed
=> Students would benefit from:

1) more practice (additional exercises and explanations) before attempting this exercise
2) scaffolding towards the intended solution (which was far simpler)

18 of 19

Conclusion

A new method to group and differentiate programming exercises based on minimal primitive construct sets, automatically extracted from successful student submissions’ abstract syntax trees
Two distance measures to quantify the number of new primitives required to complete the next exercise, having completed the previous exercise and all previous exercises
Can be used to:

Visualize the similarity between exercises (map of primitive constructs) and cluster similar exercises -> improve the exercise sequence, recommend exercises with a small increase of knowledge for practice/revision
Find key points where more practice and guidance would be beneficial
Identify exercises with overly complex solutions, including the use of new, not studied constructs, that would benefit from providing better scaffolding and intervention

19 of 19

Future work

The set of minimal primitive constructs was surprisingly large, especially for Intermediate, due to the diversity of approaches taken by students, perhaps due to using external resources
Refine our method to exclude less frequent constructs and provide a tighter and more representative set -> will improve the grouping of exercises -> recommend new exercises with few new constructs
Go beyond primitive constructs and extract more complex constructs - nested loops, recursion