2 of 11

The Problem: Solving vs. Learning

AI Copilots Solve Problems

Today's AI coding assistants (ChatGPT,

Copilot, Claude) are incredibly powerful at

writing code.

They can instantly solve competitive programming problems that challenge even experienced developers.

But Solving is Not Learning

When students use them for help, they often

just copy the answer. The assignment gets

done, but the student hasn't learned.

The Pedagogical Paradox: The same AI that could be the best tutor ever instead becomes a shortcut that undermines learning.

[

Can we build an AI tutor that is smart enough to help,�but disciplined enough to never give away the answer?

]

3 of 11

Our Approach: Guide, Don't Solve

SHARP stands for Socratic Hinting for AI-Reinforced Programming.� The core idea: break the problem into small stepping stones (milestones), track where the student is, and give them just enough of a nudge to take the next step themselves.

Adaptive Hint Scheduler

Decides when to offer help (triggered by failed test cases, time stuck, or student request).

Socratic Hint Generator

Decides what to say (always a guiding question, never the direct answer).

Student Progress Tracker

Figures out where the student currently is on the problem's roadmap.

4 of 11

System Architecture

Adaptive Hint Scheduler

Monitors student activity through the database, code execution API, and Q&A interface. When it detects the student needs help, it triggers the generator.

Socratic Hint Generator

Takes the problem description, decomposes it into milestones, retrieves relevant context, and produces a Socratic question that nudges the student toward the next milestone.

Continuous Feedback Loop

The Student Status feeds back into the scheduler, creating a closed loop. The system continuously adapts to where the student is and what they need next.

5 of 11

Milestone Division: Creating a Roadmap

The Idea

Before the tutoring session begins, we analyze the problem and create a sequence of intermediate goals. For example, if a problem has 3 bugs, the milestones might be:

MILESTONE 1

Fix the loop condition

MILESTONE 2

Fix the variable initialization

MILESTONE 3

Fix the return value

Why This Matters

Finding the best set of milestones is computationally hard. We use beam search to explore multiple possible roadmaps simultaneously and pick the optimal path.

[

The roadmap is pre-computed.�During tutoring, we only check where the student is.

]

6 of 11

Progress Estimation: Locating the Student

The core question: "Which milestone is the student's current code closest to?"

Signal 1: Semantic Similarity

We convert the student's code and each milestone's reference solution into numerical vectors (CodeBERT embeddings). Then we measure how similar they are in meaning.

Signal 2: Test-Case Pass Rate

We run the student's code against each milestone's test cases and compute what fraction they pass. This gives us an objective, functional measure of correctness for each step.

[

The milestone with the highest combined score is where the student is.

This precise localization allows the system to generate a hint that targets exactly the next step.

]

7 of 11

AskCodey: Our Open-Source Implementation

Code Editor

Students write and edit their code directly in the browser.

Test Case Feedback

Immediate pass/fail results after each code submission.

Socratic Hint Banner

When the student struggles, a guiding question appears instead of the direct answer.

Flexible LLM Backend

Supports Claude 3.5 Sonnet with GPT-4o fallback for high availability.

[

Open-source: github.com/Snail664/coding-tutor

]

8 of 11

Experiment Setup

[

We use a weaker AI model to simulate a student who struggles with code.�We then test how well different tutoring approaches guide this student.

]

Dataset

MULTI-DEBUG: 50 LeetCode problems with 1–3 injected bugs (150 scenarios).

Student Model

Ministral-8B: a less-capable AI chosen to mimic novice behavior.

Protocol & Evaluation

Max 10 turns per bug. We measure success rate, turns, indirectness, and hint relevance.

Comparison Models

No Guidance

Student tries alone with only pass/fail feedback.

Basic Guidance

Standard LLM hints without pedagogical constraints.

TreeInstruct

Socratic questions, stateless (no code verification).

SHARP (w.o. Milestone)

Our system without the step-by-step roadmap.

SHARP (Full)

Complete system with milestone estimation.

9 of 11

Results: Success Rates and Indirectness

Performance by Bug Complexity

Bugs

Method

Success

Turns

Indir.

Relev.

1 Bug

No Guidance

0.74

—

SHARP (Full)

0.98

2.60

1.0

4.94

3 Bugs

No Guidance

0.68

—

Basic Guidance

0.66

2.19

1.0

4.68

TreeInstruct

0.72

1.04

1.0

4.80

SHARP (w.o. MS)

0.88

1.31

1.0

4.84

SHARP (Full)

0.92

1.84

1.0

4.94

3-Bug Success Rates

[

98% Success Rate

Nearly every student gets guided to the correct solution on 1-bug problems.

Perfect Indirectness (1.0)

The system maintains discipline and never gives away the answer directly.

Milestone Estimation Matters

Without the roadmap, success drops from 98% to 82% on 1-bug problems.

TreeInstruct is Fast but Fragile

It uses fewer turns, but has lower success because it never verifies actual code.

]

10 of 11

Why Stateless Tutoring Fails

Problem:

Add Two Numbers (linked list). Student code has bugs in

nodes_sum

initialization and list traversal.

TreeInstruct (Stateless)

TUTOR

"What type should nodes_sum be to allow addition?"

STUDENT

"It should be an integer."

TUTOR

"Which operator gives the remainder after dividing by 10?"

RESULT

↑ No acknowledgement. Jumps topic; bug persists.

SHARP (Stateful)

TUTOR

"How is nodes_sum initialized and used in the loop?"

STUDENT

"Initialized as None; will error. Also use l2 = l2.next for

traversal."

TUTOR

"Show the fixed lines for nodes_sum and l2 update."

RESULT

↑ Acknowledges and asks for code proof; bug resolved.

11 of 11

Summary and Future Directions

Key Insights

AI tutors should guide, not solve, to preserve learner agency while providing proactive scaffolding.

Milestone-based progress tracking enables precise, focused hints that target exactly the next step.

Stateful verification, demanding code proof, is what makes Socratic tutoring actually work in practice.

Future Directions

Adaptive pacing based on frustration signals

Hierarchical milestones for complex problems

Controlled K-12 classroom deployment

Formal student modeling for mastery prediction

github.com/Snail664/coding-tutor

coding-tutor-psi.vercel.app

[

Thank you for listening�Any questions?

]