Ontology-Guided Knowledge Component Extraction

for Python Programming Exercises

Rully Agus Hendrawan, Shravya Suresh, Rafaella Sampaio de Alencar,

Alice Micheli, Peter Brusilovsky, Sergey Sosnovsky

CSEDM 2026 - 10th Workshop on Educational Data Mining in Computer Science Education

Slide 1

1

Ontology-Guided Knowledge Component

Extraction for Python Programming Exercises

1. Background

2. Method

3. Evaluation & Results

4. Takeaways

Slide 2

2

Adaptive systems need stable KC

Slide 3

3

Programming repositories grow faster than expert annotation workflows.

Manual KC labeling is interpretable, but costly and hard to scale.

Open-ended LLM labels vary in wording with semantic invariance, inconsistent granularity, and coverage.

Content

→

KC

→

Learner model

stable

same concept vocabulary across activities and time

scalable

annotation can keep pace with new exercises

Linking content → knowledge components (KC)

Slide 4

4

Open-ended generation

Ontology-guided annotation

LLM invents labels

Label wording drifts

Harder to connect to curricula

Expert ontology defines valid KC

LLM selects and scores fixed candidates

Outputs map to reusable learner models

Ontology-Guided Knowledge Component

Extraction for Python Programming Exercises

1. Background

2. Method

3. Evaluation & Results

4. Takeaways

Slide 2

2

The study: Dataset

Slide 5

5

262

Python learning activities

128

ontology classes

98

leaf-level candidate KCs

CLAWS for Concept Extraction & Labelling (Multi-Label Classification via Aggregation and Window Sliding)

7

Large Language Model

3

repeated batches when feasible

Window, batch, and threshold are calibration parameters

Slide 8

8

KC-window size (w)

5, 10, 20, or no-window

controls candidate-label context size

Batch count

three repeats when feasible

handle variance through mode aggregation

Threshold (t)

3, 4, or 5

moves precision-recall and density

Models deployment trade-off: large/ medium/ small size, hosted/ local

Slide 9

9

GPT-4.1-mini

Compact generalist API

hosted, lower-cost option

DeepSeek-V3

MoE generalist API

hosted, 671B total; 37B active

Claude Haiku 4.5

Efficient generalist API

hosted, speed/cost-oriented

Llama 3.3 70B

Open-weight generalist API

more deployment control

Mistral 7B

Small generalist local

lightweight local deployment

Qwen2.5-Coder 7B

Code specialist local

code-specialized local subset

Hosted models offer strong instruction following, large context windows, and high structured-output reliability, making them attractive for offline indexing of programming exercises.

​

However, hosted APIs can be expensive and unpredictable at scale, especially for real-time student feedback where every learner interaction may trigger model calls, and latency can also be a practical barrier. Local/open-weight models offer more control over cost, privacy, availability, and deployment.

​

​

Apr 2025

Dec 2024

Oct 2025

Dec 2024

Sep 2023

Nov 2024

1M tokens

128K tokens

200K tokens

128K tokens

8K to 32K

128K tokens

Ontology-Guided Knowledge Component

Extraction for Python Programming Exercises

1. Background

2. Method

3. Evaluation & Results

4. Takeaways

Slide 10

10

Output compliance tends to be better in larger models (70B and more)

Slide 11

11

GPT-4.1-mini

99.6

% parse success

DeepSeek-V3

100.0

% parse success

Claude Haiku 4.5

99.5

% parse success

Llama 3.3 70B

99.3

% parse success

Mistral 7B

63.6

% parse success

Qwen2.5-Coder 7B

61.0

% parse success

Non-compliance:�malformed CSV, missing labels, and hallucinated concepts.

High compliance did not guarantee stable generation across KC window size

Slide 12

12

w=5

w=10

w=20

no-window

w=all=98

GPT-4.1-mini

0.92

0.91

0.90

0.71

DeepSeek-V3

0.95

0.95

0.85

0.96

Claude Haiku 4.5

0.71

0.79

0.76

0.28

Llama 3.3 70B

0.57

0.53

0.39

0.26

Mean pairwise Jaccard across repeated batches

DeepSeek-V3 stays stable across window size; �no-window choked GPT-4.1-mini, Claude, and Llama.

3

repeated batches

DeepSeek-V3 and GPT-4.1-mini delivered the strongest expert agreement

Slide 13

13

DeepSeek-V3

No window, t=4

F1 0.93 / J 0.88

13.7 KCs/ex

GPT-4.1-mini

w=5, t=4

F1 0.90 / J 0.83

15 KCs/ex

Claude Haiku 4.5

w=5, t=3

F1 0.80 / J 0.70

13.9 KCs/ex

Llama 3.3 70B

w=10, t=4

F1 0.73 / J 0.58

18.9 KCs/ex

Expected prediction density

Expert average: 13.41 concepts per exercise

​

Best KC-window size was model dependent

Slide 14

14

w=5

w=10

w=20

No window

GPT-4.1-mini

0.90

0.84

0.88

0.88

DeepSeek-V3

0.87

0.89

0.88

0.93

Claude Haiku 4.5

0.79

0.77

0.76

0.73

Llama 3.3 70B

0.71

0.73

0.70

0.67

Macro F1 at threshold t = 4.

Highlighted cells are the best setting for that model class.

Thresholds affect prediction density (w = 5)

3-batch aggregate; bars show average predicted KCs per exercise at each threshold.

Gold standard density = 13.4 KC/ex

Slide 15

15

t=3

t=4

t=5

drop from t=3 to t=5

GPT-4.1-mini

15.8

15.0

13.5

-2.30 KC/ex

DeepSeek-V3

16.3

15.6

14.6

-1.70 KC/ex

Claude Haiku 4.5

13.9

13.0

8.60

-5.3 KC/ex

Llama 3.3 70B

20.7

19.0

14.4

-6.3 KC/ex

Mistral 7B

6.17

5.48

3.86

-2.31 KC/ex

Raising t from 3 to 5 lowers prediction density for every model;

the drop is model-dependent (-1.7 to -6.3 KCs/ex).

Best configurations are models specific

Slide 20

20

Model

Best setting

Notes

DeepSeek-V3

No window, t=4: F1 .93 / J .88

strongest expert agreement and �prediction density close to gold

GPT-4.1-mini

w=5, t=4: F1 .90 / J .83

strong windowed performance; �slightly recall-oriented

Claude Haiku 4.5

w=5, t=3: F1 .80 / J .70

usable but threshold-sensitive

Llama 3.3 70B

w=10, t=4/5: F1 about .73

broader predictions; lower stability

Local 7B models

Malformed: low parse success / sparse KCs

need output control before fair deployment

Local models need output control before they are deployment-ready

Slide 16

16

Schema validation + repair

Constrained decoding or few-shot format

Recalibrate threshold and KC-window

Deploy after stable expert agreement

Mistral 7B and Qwen2.5-Coder 7B results are not ready �for the current CLAWS pipeline; regardless their promising semantic capability.

Deployment should calibrate the LLM harness for KC extraction

Slide 17

17

1

Pick candidate model class

hosted/API, open-weight, or local control requirements

2

Run expert-labeled validation

measure compliance, stability, F1/Jaccard, and density

3

Tune w and t together

windowing helps some models and hurts or adds cost for others

Prerequisite for pipeline readiness

structured output and stable repeated batches before deployment/ scale-up

CONCLUSION:

LLM-assisted Ontology-guided KC extraction works, as a calibrated pipeline

Slide 18

18

Human experts define the reusable ontology (e.g. Python); �LLMs scale exercise annotation.

Hosted generalist (representative) models were most reliable under the current CLAWS harness.

DeepSeek-V3: no-window; GPT-4.1-mini: with w=5.

Window size, threshold, and prediction density must be calibrated (with expert) when changing models.

Ontology-Guided Knowledge Component Extraction

for Python Programming Exercises

Rully Agus Hendrawan, Shravya Suresh, Rafaella Sampaio de Alencar,

Alice Micheli, Peter Brusilovsky, Sergey Sosnovsky

CSEDM 2026

10th Workshop on Educational Data Mining in Computer Science Education

Slide 1

1

Large Language Model

Thank you

Backup / Q&A

Ontology-Guided Knowledge Component Extraction for Python Programming Exercises

Slide 19

19

CLAWS pseudocode (per exercise)

22

LLM prompt (template)

23

Example model output (per candidate concept)

exercise,parent,leaf,reason,score�... ,HandlingList,ListReferencing,"assigns one list variable to another",5�... ,HandlingList,ReversingList,"no reversal operation",0�... ,HandlingList,DeletingListElement,"no deletion operation",0

24