1 of 102

1

Context Engineering in 2026: Compaction, Memory & Skills

The 2026 playbook, shown live on an open-source tutor

https://github.com/towardsai/ai-tutor-app

Louis-François Bouchard · Samridhi Vaid · Omar Solano

2 of 102

2

Every long agent session, ever.

turn 1 you: "One rule: NEVER push directly to main."

turn 2 agent: "Understood — feature branches only."

··· 45 turns of greps, diffs and logs ···

turn 47 agent: $ git push origin main

"Done! Pushed straight to main."

The model didn't get dumber. Its context did.

turn 1 you: "One rule: NEVER push directly to main."

turn 2 agent: "Understood — feature branches only."

··· 45 turns of greps, diffs and logs ···

turn 47 agent: $ git push origin main

"Done! Pushed straight to main."

turn 48 you: "NO FFS I TOLD YOU NOT TO F***ING DO IT. Switching to claude"

3 of 102

3

4 of 102

4

how to fix “context rot”

(in our AI tutor)

5 of 102

5

3

The next 80 minutes

Part 1 — Compaction, memory, retrieval, skills…

Louis-François — the techniques + why the tutor needs them

Part 2 — The architecture + the eval harness, on Gemini 3.5 Flash

Omar — system design + measuring tokens, cost, latency, caching…

Part 3 — How to match the quality at a fraction of the cost

Samridhi — open & local models, experiments, compaction and how we chose

6 of 102

6

4

Hi — From Towards AI

Louis-François Bouchard

Co-founder & CTO @ Towards AI (2020-)�"What's AI" on YouTube (2020)�Author of Building LLMs for Production (2024)�ex-Mila PhD(c) (2022-2024)

Samridhi Vaid

Senior ML Engineer @ Towards AI

M.Sc Computer Science

Omar Solano

Senior ML Engineer @ Towards AI

B.Eng. in Automated Production Engineering

7 of 102

7

Towards AI (Academy)

We build courses for AI engineers…

… and provide an AI tutor for them!

we build lessons, newsletters, videos & trainings for a living — lots of it runs on agents that must not forget

8 of 102

8

6

The case study: our AI tutor

We had five requirements…

answers grounded in our content, not general knowledge
stay in student's current lesson, not from any course we have
hold long help sessions: debugging, follow-ups, “explain that again”
handle code, read the student's, generate working examples
stream fast, latency is UX for a tutor

→ context engineering (in 2026)

9 of 102

9

7

Problems…

the context window is finite — everything the model sees competes for one attention budget

Instructions, retrieved lessons, tool use, code…

the model is stateless — every call starts from zero; between sessions, nothing persists

→ within a session: context management across sessions: memory

10 of 102

10

8

What the model actually sees, every call

system

prompt

tool

defs

chat history

old tool outputs

course chunks

+ memory

the student's question

the window

(old) old lesson chunks

tool-call + tool-result pairs

searches + files it opened

$

$$

$$$

11 of 102

11

9

Long context hurts — quality

quality degrades as the window fills — models lose the middle: “context rot”
lost in the middle — recall sags for facts buried deep in the window

12 of 102

12

11

Long context hurts — cost & latency

the whole history is re-sent every turn — you re-pay for every token, every turn
TTFT (time to first token) climbs with input size — the model reprocesses the entire window each call (latency)

→ manage it for spend & speed — not because quality drops

13 of 102

13

12

Compaction:

keep what still matters.

Drop or relocate the rest.

14 of 102

14

13

Start cheap — no LLM call required

Observation truncation (outliers) — cut one huge tool output to head + tail before it enters history�when: one result is huge but only the input and outcome matter — a student's 300-line stack trace (fires the moment one huge output arrives)
Trim / sliding window — drop the oldest turns — or keep the last N, with overlap for continuity�when: old turns no longer matter — previous, unrelated bugs or questions (continuous)
Tool-result clearing (for specific verbatim tools) — swap old outputs for a re-fetchable placeholder, keep the call record�when: lots of temporary outputs dominate — stale course-chunk retrievals; the safest first move (fires later, once an output is stale)

Trivial tools → Spend to save → Offload

15 of 102

15

14

Then: spend tokens to save tokens (LM-based)

Selective retention — keep constraints, decisions, open tasks; drop exploration & duplicates�when: you can judge what matters — keep the student's goal & env, drop dead ends
Summarization — continuously replace old history with a running gist — lossy by design�when: the gist is enough — what the student already understood, not every word
Compaction (summary + reset) — collapse the whole history into one fresh summary, then start over from it (e.g. when claude code reaches context limit)�careful: spend-to-save can backfire

Trivial tools → Spend to save → Offload

16 of 102

16

15

Spend tokens to save tokens — the full catalog (1/2)

Trivial tools → Spend to save → Offload

17 of 102

17

16

…or move tokens out of the window (2/2)

→ GraphRAG $$/course index — ties plain RAG

Trivial tools → Spend to save → Offload

18 of 102

18

17

Or don't delete — relocate

Offload to files / memory — persist details to disk or a memory tool; keep a pointer in context
reversible — nothing is lost; retrieval pulls it back in just-in-time
Karpathy’s "LLM wiki" — a file tree the agent maintains & re-reads: CLAUDE.md, AGENTS.md, a growing KB

+ RAG DB!

→ the file system becomes cheap, durable, inspectable context

Trivial tools → Spend to save → Offload

19 of 102

19

18

A recipe from production

session/

01-auth-research.md chunk

02-db-decisions.md links to 01

index.md the map

1- write chunks to files, cross-linked with pointers

2- keep one index file that maps them all

3- reset the agent — only the index goes back in

4- it searches its way back to details, per task, limiting context pollution

→ kilobytes in context — agent pulls context according to task complexity

the same shape Claude Code & friends converged on: durable files + a small live window.

Trivial tools → Spend to save → Offload

20 of 102

20

35

Skills Parenthesis: load instructions like code

more small, focused skills — explain-concept · evaluation · review-my-code — not one giant file
progressive disclosure — names/small desc. always visible, bodies load only on use

Trivial tools → Spend to save → Offload

21 of 102

21

12

Compaction:

keep what still matters.

Drop or relocate the rest.

22 of 102

22

20

BUT, there’s a problem with compaction methods…

Image from https://x.com/its_ao/status/2070556265906917860 (amazing read!)

23 of 102

23

20

BUT, there’s a problem with compaction methods…

Image from https://x.com/its_ao/status/2070556265906917860 (amazing read!)

24 of 102

24

20

Caching: why it works (and what you pay)

25 of 102

25

19

Caching: a game-changer

implicit: automatic, default-on — Gemini 2.5+ (~90% off), DeepSeek ($0.14→$0.0028/M ≈50×)...
explicit: you pin it — Anthropic cache_control / Gemini cache: guaranteed, but TTL + storage rent

Not only helps with cost but also greatly reduces latency!

→ summarization (potentially) is a trap: rewriting the prefix throws the cache away

�As Manus AI puts it:

”If I had to choose just one metric, I'd argue that the KV-cache hit rate is the single most important metric for a production-stage AI agent.”

26 of 102

26

21

2026: every serious harness converged here

Claude Code — five mechanisms: microcompact (no LLM) trims tool outputs every turn; near ~95% a 9-section structured summary replaces history; files/skills/plan re-injected after�the summary is readable & editable — and CLAUDE.md survives compaction
Codex — one move, summarize → replace: a local LLM writes a 'handoff summary' (prompt is in the codex-rs repo); for OpenAI models /responses/compact returns an encrypted blob instead�fires pre-turn AND mid-turn — and tries structured session memory first, a full LLM summary only if that's not enough
the APIs themselves — Anthropic & OpenAI both ship it server-side: set context_management → a compaction block comes back, old turns dropped, system prompt stays cached�same model writes the summary — accurate recall matters more than speed or cost

27 of 102

27

21

2026: every serious harness converged here

They all don't just compact, so shouldn’t you.

Start fresh sessions when switching tasks.
Scope file context narrowly.
Disconnect unused tools.
Compact.
Optimize for cache hits. (Model providers improve their feature set for prompt caching frequently)
User a model router based on task complexity.

Log everything (e.g. LangSmith, Opik…) and monitor what’s happening (cache hit rate, user frustration, abnormally long outputs…)

Setting explicit cache breakpoints when supported
Opting in to provider-side implicit caching when explicit breakpoints aren’t supported
Structuring your prompt to maximize cache reads

// In Deep Agents you get prompt caching for free!

const agent = createDeepAgent({

model: "gpt-5.5",

});

// In LangChain, opt in via our middleware:

const agent = createAgent({

model: "claude-haiku-4-5-20251001",

middleware: [anthropicPromptCachingMiddleware()],

});

claude-haiku-4-5: -77%. Using Anthropic's explicit breakpoints, we can keep a large portion of the prompt cached. This significantly reduced the token cost of each request.
gpt-5.4-mini: -80%. OpenAI's automatic longest-prefix caching gives us a sizable 80% cost reduction
gemini-3.5-flash: -49%. Gemini's implicit caching makes no explicit savings guarantee, but we still see considerable savings

For each invocation you get time-to-first-token, total input tokens, cache-read tokens, and total output tokens rolled up to a per-trajectory aggregate

28 of 102

28

36

All of it together is context engineering:

deciding what the model sees, every single call.

prompt engineering grew up: compaction + memory + skills under one discipline.

29 of 102

29

30

The pattern we engineer toward

keep in the window (optimized with caching)

the student's goal, their environment (current course, lesson, current code setup/repo), decisions made, the current error…

drop or compact (using various techniques)

old stack traces → truncate · dead-end attempts → selective retention · chit-chat → trim…

persist across sessions (memory)

level & progress · the session's gist…

Follow along!

30 of 102

30

PART 2

The tutor, built & measured

system design · the eval harness · Gemini 3.5 baseline

Follow along!

31 of 102

31

One request, end to end

ONE AGENT

LangChain create_agent + InMemorySaver

Gemini 3.5 Flash

32 of 102

32

One request, end to end

ONE AGENT

LangChain create_agent + InMemorySaver

Gemini 3.5 Flash

middleware stack

summarize · clear tool-outputs · source preference

retrieve_tutor_context

hybrid RAG

run_kb_command

browse the KB

33 of 102

33

One request, end to end

ONE AGENT

LangChain create_agent + InMemorySaver

Gemini 3.5 Flash

middleware stack

summarize · clear tool-outputs · source preference

retrieve_tutor_context

hybrid RAG

run_kb_command

browse the KB

Next.js UI

browser chat

FastAPI

POST /api/chat

request

34 of 102

34

One request, end to end

ONE AGENT

LangChain create_agent + InMemorySaver

Gemini 3.5 Flash

middleware stack

summarize · clear tool-outputs · source preference

retrieve_tutor_context

hybrid RAG

run_kb_command

browse the KB

Next.js UI

browser chat

FastAPI

POST /api/chat

request

stream

→ the agent is small. The grounding, our course and docs knowledge base, is what makes it a tutor.

35 of 102

35

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

36 of 102

36

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

the question

from a student in a course

37 of 102

37

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

the question

from a student in a course

scope to source

selected source only, improves recall

38 of 102

38

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

the question

from a student in a course

scope to source

selected source only, lifts precision

hybrid search

dense embed-v4 top 15 + BM25 top 30

39 of 102

39

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

the question

from a student in a course

scope to source

selected source only, lifts precision

hybrid search

dense embed-v4 top 15 + BM25 top 30

fuse + rerank

RRF to merge to 45, Cohere rerank to top 5, drop <0.10

40 of 102

40

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

the question

from a student in a course

scope to source

selected source only, lifts precision

hybrid search

dense embed-v4 top 15 + BM25 top 30

fuse + rerank

RRF to merge to 45, Cohere rerank to top 5, drop <0.10

token budget

fill up to 100k, into the window

41 of 102

41

Grounding tool 1: retrieve_tutor_context

Corpus: 8.62M tokens, 3,203 docs, 14 sources (5 courses + 9 doc sets).

No window holds it, so we embed once and retrieve per question.

the question

from a student in a course

scope to source

selected source only, lifts precision

hybrid search

dense embed-v4 top 15 + BM25 top 30

fuse + rerank

RRF to merge to 45, Cohere rerank to top 5, drop <0.10

token budget

fill up to 100k, into the window

→ top-k is precise but narrow. What if the answer is spread across the corpus? Or if agent benefits from more search granularity?

https://arxiv.org/pdf/2605.05242

42 of 102

42

Grounding tool 2: run_kb_command, browse the KB

raw/

3,203 markdown mirrors (254 course + 2,949 docs)

generated/

machine indexes it greps: doc manifest, headings, code symbols

wiki/

29 pages: 11 topics, 9 frameworks, 5 courses

Grounding tool 2, run_kb_command: a sandboxed read-only shell over data/kb/. Limits: 8s timeout, 40,000-char output cap, max 20 commands/turn, no network/writes, jailed to data/kb/.
Three layers: raw/ = 3,203 markdown mirrors (254 course + 2,949 docs);

generated/ = machine indexes the agent greps instead of scanning everything (a manifest = one row per document with path/title/token-count/url; a headings index = every markdown heading with its line; a code-symbol index = every function/class name mapped to where it appears);

wiki/ = 29 short pages an LLM maintains (Karpathy's 'LLM wiki' idea): 11 topics (concepts that cross sources, like rag/agents/fine-tuning), 9 frameworks (one library each), 5 courses. The maintainer builds the wiki offline via MAINTAINER.md; the tutor only reads it. Browse is the main grounding path (~89% of turns). The retrieve-only comparison is the KB-off arm, which you'll see in the results: it grounds just as well but costs more and remembers less.

43 of 102

43

Grounding tool 2: run_kb_command, browse the KB

A read-only shell over the corpus: rg, grep, find, ls, sed, head, cat, wc.

raw/

3,203 markdown mirrors (254 course + 2,949 docs)

generated/

machine indexes it greps: doc manifest, headings, code symbols

wiki/

29 pages: 11 topics, 9 frameworks, 5 courses

Grounding tool 2, run_kb_command: a sandboxed read-only shell over data/kb/. Limits: 8s timeout, 40,000-char output cap, max 20 commands/turn, no network/writes, jailed to data/kb/.
Three layers: raw/ = 3,203 markdown mirrors (254 course + 2,949 docs);

generated/ = machine indexes the agent greps instead of scanning everything (a manifest = one row per document with path/title/token-count/url; a headings index = every markdown heading with its line; a code-symbol index = every function/class name mapped to where it appears);

wiki/ = 29 short pages an LLM maintains (Karpathy's 'LLM wiki' idea): 11 topics (concepts that cross sources, like rag/agents/fine-tuning), 9 frameworks (one library each), 5 courses. The maintainer builds the wiki offline via MAINTAINER.md; the tutor only reads it. Browse is the main grounding path (~89% of turns). The retrieve-only comparison is the KB-off arm, which you'll see in the results: it grounds just as well but costs more and remembers less.

44 of 102

44

Grounding tool 2: run_kb_command, browse the KB

A read-only shell over the corpus: rg, grep, find, ls, sed, head, cat, wc.
Limits: 8-second timeout, output capped at 40,000 chars, max 20 commands/turn, sandboxed to data/kb/.

raw/

3,203 markdown mirrors (254 course + 2,949 docs)

generated/

machine indexes it greps: doc manifest, headings, code symbols

wiki/

29 pages: 11 topics, 9 frameworks, 5 courses

Grounding tool 2, run_kb_command: a sandboxed read-only shell over data/kb/. Limits: 8s timeout, 40,000-char output cap, max 20 commands/turn, no network/writes, jailed to data/kb/.
Three layers: raw/ = 3,203 markdown mirrors (254 course + 2,949 docs);

generated/ = machine indexes the agent greps instead of scanning everything (a manifest = one row per document with path/title/token-count/url; a headings index = every markdown heading with its line; a code-symbol index = every function/class name mapped to where it appears);

wiki/ = 29 short pages an LLM maintains (Karpathy's 'LLM wiki' idea): 11 topics (concepts that cross sources, like rag/agents/fine-tuning), 9 frameworks (one library each), 5 courses. The maintainer builds the wiki offline via MAINTAINER.md; the tutor only reads it. Browse is the main grounding path (~89% of turns). The retrieve-only comparison is the KB-off arm, which you'll see in the results: it grounds just as well but costs more and remembers less.

45 of 102

45

Grounding tool 2: run_kb_command, browse the KB

A read-only shell over the corpus: rg, grep, find, ls, sed, head, cat, wc.
Limits: 8-second timeout, output capped at 40,000 chars, max 20 commands/turn, sandboxed to data/kb/.
Built offline, read at runtime: an agent maintains the wiki via data/kb/MAINTAINER.md when sources change; the tutor only reads it.

raw/

3,203 markdown mirrors (254 course + 2,949 docs)

generated/

machine indexes it greps: doc manifest, headings, code symbols

wiki/

29 pages: 11 topics, 9 frameworks, 5 courses

Grounding tool 2, run_kb_command: a sandboxed read-only shell over data/kb/. Limits: 8s timeout, 40,000-char output cap, max 20 commands/turn, no network/writes, jailed to data/kb/.
Three layers: raw/ = 3,203 markdown mirrors (254 course + 2,949 docs);

generated/ = machine indexes the agent greps instead of scanning everything (a manifest = one row per document with path/title/token-count/url; a headings index = every markdown heading with its line; a code-symbol index = every function/class name mapped to where it appears);

wiki/ = 29 short pages an LLM maintains (Karpathy's 'LLM wiki' idea): 11 topics (concepts that cross sources, like rag/agents/fine-tuning), 9 frameworks (one library each), 5 courses. The maintainer builds the wiki offline via MAINTAINER.md; the tutor only reads it. Browse is the main grounding path (~89% of turns). The retrieve-only comparison is the KB-off arm, which you'll see in the results: it grounds just as well but costs more and remembers less.

46 of 102

46

Grounding tool 2: run_kb_command, browse the KB

raw/

3,203 markdown mirrors (254 course + 2,949 docs)

generated/

machine indexes it greps: doc manifest, headings, code symbols

wiki/

29 pages: 11 topics, 9 frameworks, 5 courses

→ With our current system prompt, browse is the main grounding path (~89% of turns).

A read-only shell over the corpus: rg, grep, find, ls, sed, head, cat, wc.
Limits: 8-second timeout, output capped at 40,000 chars, max 20 commands/turn, sandboxed to data/kb/.
Built offline, read at runtime: an agent maintains the wiki via data/kb/MAINTAINER.md when sources change; the tutor only reads it.

Grounding tool 2, run_kb_command: a sandboxed read-only shell over data/kb/. Limits: 8s timeout, 40,000-char output cap, max 20 commands/turn, no network/writes, jailed to data/kb/.
Three layers: raw/ = 3,203 markdown mirrors (254 course + 2,949 docs);

generated/ = machine indexes the agent greps instead of scanning everything (a manifest = one row per document with path/title/token-count/url; a headings index = every markdown heading with its line; a code-symbol index = every function/class name mapped to where it appears);

wiki/ = 29 short pages an LLM maintains (Karpathy's 'LLM wiki' idea): 11 topics (concepts that cross sources, like rag/agents/fine-tuning), 9 frameworks (one library each), 5 courses. The maintainer builds the wiki offline via MAINTAINER.md; the tutor only reads it. Browse is the main grounding path (~89% of turns). The retrieve-only comparison is the KB-off arm, which you'll see in the results: it grounds just as well but costs more and remembers less.

47 of 102

47

Grounding tool 2: run_kb_command, browse the KB

raw/

3,203 markdown mirrors (254 course + 2,949 docs)

generated/

machine indexes it greps: doc manifest, headings, code symbols

wiki/

29 pages: 11 topics, 9 frameworks, 5 courses

→ With our current system prompt, browse is the main grounding path (~89% of turns).

→ Turning run_kb_command OFF = Tutor is 50% faster (~19sec vs ~38sec) and the same retrieval recall.

A read-only shell over the corpus: rg, grep, find, ls, sed, head, cat, wc.
Limits: 8-second timeout, output capped at 40,000 chars, max 20 commands/turn, sandboxed to data/kb/.
Built offline, read at runtime: an agent maintains the wiki via data/kb/MAINTAINER.md when sources change; the tutor only reads it.

Grounding tool 2, run_kb_command: a sandboxed read-only shell over data/kb/. Limits: 8s timeout, 40,000-char output cap, max 20 commands/turn, no network/writes, jailed to data/kb/.
Three layers: raw/ = 3,203 markdown mirrors (254 course + 2,949 docs);

generated/ = machine indexes the agent greps instead of scanning everything (a manifest = one row per document with path/title/token-count/url; a headings index = every markdown heading with its line; a code-symbol index = every function/class name mapped to where it appears);

wiki/ = 29 short pages an LLM maintains (Karpathy's 'LLM wiki' idea): 11 topics (concepts that cross sources, like rag/agents/fine-tuning), 9 frameworks (one library each), 5 courses. The maintainer builds the wiki offline via MAINTAINER.md; the tutor only reads it. Browse is the main grounding path (~89% of turns). The retrieve-only comparison is the KB-off arm, which you'll see in the results: it grounds just as well but costs more and remembers less.

48 of 102

48

How the tutor manages its own context

the production preset: a stack of 3 middlewares

clear stale tool-outputs

at 5k, keeps retrieval and latest 5

summarize old turns

at 30k, keep last 20

source preference

scope the corpus

49 of 102

49

How the tutor manages its own context

the production preset: a stack of 3 middlewares

clear stale tool-outputs

at 5k, keeps retrieval and latest 5

summarize old turns

at 30k, keep last 20

source preference

scope the corpus

→ Production with unproven defaults. Which configuration is actually best?

50 of 102

50

Demo: the tutor, live

https://huggingface.co/spaces/towardsai-tutors/context-engineering-experiments

51 of 102

51

Demo: the tutor, live

https://huggingface.co/spaces/towardsai-tutors/context-engineering-experiments

What we log per turn (telemetry)

input / output tokens · cached

estimated $ · time-to-first-token
model calls · did compaction fire?

52 of 102

52

Why measure at all?

The field's advice is confident, but as we discovered, under modern prompt caching the obvious move can invert.

53 of 102

53

Why measure at all?

The field's advice is confident, but as we discovered, under modern prompt caching the obvious move can invert.
The space is too big to eyeball: ~many configurations across quality, memory, tokens, cost, latency.

54 of 102

54

Why measure at all?

The field's advice is confident, but as we discovered, under modern prompt caching the obvious move can invert.
The space is too big to eyeball: ~many configurations across quality, memory, tokens, cost, latency.

→ so we built a harness that runs the real agent and grades it offline. First, the vocabulary.

55 of 102

55

The vocabulary

→ for the Gemini study: multiple presets, all on gemini-3.5-flash. What do we test on?

Preset: a specific setup for the AI tutor we tested, everything else held fixed
Task type: one test set for one specific task.

We use two in this study: single-turn, sessions

run: one preset × one model × one task type

E.g. hybrid search x Gemini 3.5 x single-turn

bundle: the saved JSON for one turn: answer, tools, sources, tokens, timing, did-compaction-fire.

Vocabulary, keep it light.

CLEAN MENTAL MODEL: a battery is a task suite stored as one JSONL.
A run is one battery executed with one model and one arm.
A bundle is one saved tutor turn (question, answer, tools, sources, tokens, cost, timing, errors).
An arm is one experimental CONDITION: sometimes just a memory preset (full_history), sometimes prod plus a CLI change (prod+kb_off, prod+retrieval_budget_30k).
A preset is a named middleware config; 15 exist in code. We tested 13 arms: 11 are presets, 2 (kb_off, retrieval_budget_30k) are prod + a CLI flag. That is the 15-vs-13 gap.
On screen we lead with arm (13 total); preset is just the code name. The 4 presets we did NOT run on Gemini: summarization_only, editing_only, delta_summarization, hierarchical_summarization. Three of them (summarization_only, delta_summarization, hierarchical_summarization) reappear in Part 3 as the local-model 'summarize' arms; editing_only is unused. Replay was built but never run (dropped from the slide).

56 of 102

56

What we test on: tasks from real students

single-turn

60 questions, asked once

measures:

- retrieval (auto graded)

- right facts (key points)

- right kind of response (teach / redirect)

ex: “repo only has lessons

4,5,6,8,10, is that expected?”

From 151 real academy posts; we kept 60 after cleaning/reviewing.

What we test on, three tasks each with its own dataset.

The four measures:

single-turn measures right facts (key points) + right kind of response (behavior: teach / redirect-to-support / acknowledge-feedback);
sessions measure memory probes under compaction;
personas measure profile personalization;

Composition (verified):
single-turn 60 = 37 answer_from_corpus + 10 redirect_to_support + 7 answer_general + 6 acknowledge_feedback.
Built from 151 real academy posts, kept 60 after review (cut 32, mostly stale).
Code/auto graded:�Retrieval (retrieval_metrics)

Citations (citation_metrics)�Behavior proxy (behavior_heuristic)

Telemetry

LLM judge�Key-point coverage “does the answer convey or clearly imply this atomic claim?”
Behavior = "right kind of response"

�

Sessions: 3 used (s01=13, s02=11, s08=12 turns = 36 turns/arm/trial), 4 graded probes each, so 12 probes/arm in the broad screen (1 trial) and 24 in the deeper bake-off (2 trials).
planted_facts — synthetic student facts we invent
The middle part, the filler/context turns are drawn from the same real-post pool the single-turn 60 came from.
Probe is the question from the student, that needs the planted fact
The pass/fail rule to the judge: "Must pick RAG evaluation and name hit rate + MRR; any other topic or metrics = fail."
Probe types:
fact_recall = the answer must use a fact the student stated many turns earlier (e.g. "RAG eval, hit rate, MRR").
preference_compliance = the tutor must honor a stated preference (OS/tooling/language/style, e.g. conda not venv). So "fact_recall 83% vs prod 33%" means: of the fact_recall probes (n=6 in that rerun), profile_memory passed 83%, prod 33%.
Every probe is judge graded

Personas: 10 profiles x 4 questions = 40.

GRADING - auto (code) vs validated LLM judge (blinded to the arm).
Rule: code computes what it reliably can; the judge handles nuance. Judge validated at 98% agreement / TPR 100% / TNR 96% vs human on 96 probes.

single-turn: AUTO = retrieval recall, citation match, behavior proxy, telemetry. JUDGE = key-point coverage (right facts), behavior (right kind of response), holistic, faithfulness.

sessions: AUTO = telemetry (cost/latency/tokens) + the gate (check_triggers: did compaction fire before the probe). JUDGE = the probe verdict (did it recall the planted fact) + holistic/faithfulness.

personas: AUTO = regex_any checks + anti-patterns (auto-fail). JUDGE = only the nuanced 'llm' checks. auto_pass = fail if an anti-pattern hits, else pass if all regex pass, else send to the judge.
PERSONAS in plain terms: each persona is a student profile (e.g. uses conda) seeded into long-term memory (here its actually just the inmemory setup); we ask 4 questions on a FRESH thread so only the stored profile carries the info, then check the answer reflects it - regex must-haves (e.g. 'conda'), anti-patterns it must avoid (e.g. 'venv'), plus optional llm checks for nuance. Measures long-term profile memory; off in production.

(seed → store by student_id → inject into system prompt → fresh thread

57 of 102

57

What we test on: tasks from real students

single-turn

60 questions, asked once

measures:

- retrieval (auto graded)

- right facts (key points)

- right kind of response (teach / redirect)

ex: “repo only has lessons

4,5,6,8,10, is that expected?”

sessions (multi-turns)

11 to 13 turns (more turns in part 3)

measures:

- in-session memory

under compaction

ex: plant “I want to learn about RAG”

at turn 0, probe it at turn 11

Invented facts, middle filler messages from real academy posts;

Most interesting task type for our Gemini study

What we test on, three tasks each with its own dataset.

The four measures:

single-turn measures right facts (key points) + right kind of response (behavior: teach / redirect-to-support / acknowledge-feedback);
sessions measure memory probes under compaction;
personas measure profile personalization;

Composition (verified):
single-turn 60 = 37 answer_from_corpus + 10 redirect_to_support + 7 answer_general + 6 acknowledge_feedback.
Built from 151 real academy posts, kept 60 after review (cut 32, mostly stale).
Code/auto graded:�Retrieval (retrieval_metrics)

Citations (citation_metrics)�Behavior proxy (behavior_heuristic)

Telemetry

LLM judge�Key-point coverage “does the answer convey or clearly imply this atomic claim?”
Behavior = "right kind of response"

�

Sessions: 3 used (s01=13, s02=11, s08=12 turns = 36 turns/arm/trial), 4 graded probes each, so 12 probes/arm in the broad screen (1 trial) and 24 in the deeper bake-off (2 trials).
planted_facts — synthetic student facts we invent
The middle part, the filler/context turns are drawn from the same real-post pool the single-turn 60 came from.
Probe is the question from the student, that needs the planted fact
The pass/fail rule to the judge: "Must pick RAG evaluation and name hit rate + MRR; any other topic or metrics = fail."
Probe types:
fact_recall = the answer must use a fact the student stated many turns earlier (e.g. "RAG eval, hit rate, MRR").
preference_compliance = the tutor must honor a stated preference (OS/tooling/language/style, e.g. conda not venv). So "fact_recall 83% vs prod 33%" means: of the fact_recall probes (n=6 in that rerun), profile_memory passed 83%, prod 33%.
Every probe is judge graded

Personas: 10 profiles x 4 questions = 40.

GRADING - auto (code) vs validated LLM judge (blinded to the arm).
Rule: code computes what it reliably can; the judge handles nuance. Judge validated at 98% agreement / TPR 100% / TNR 96% vs human on 96 probes.

single-turn: AUTO = retrieval recall, citation match, behavior proxy, telemetry. JUDGE = key-point coverage (right facts), behavior (right kind of response), holistic, faithfulness.

sessions: AUTO = telemetry (cost/latency/tokens) + the gate (check_triggers: did compaction fire before the probe). JUDGE = the probe verdict (did it recall the planted fact) + holistic/faithfulness.

personas: AUTO = regex_any checks + anti-patterns (auto-fail). JUDGE = only the nuanced 'llm' checks. auto_pass = fail if an anti-pattern hits, else pass if all regex pass, else send to the judge.
PERSONAS in plain terms: each persona is a student profile (e.g. uses conda) seeded into long-term memory (here its actually just the inmemory setup); we ask 4 questions on a FRESH thread so only the stored profile carries the info, then check the answer reflects it - regex must-haves (e.g. 'conda'), anti-patterns it must avoid (e.g. 'venv'), plus optional llm checks for nuance. Measures long-term profile memory; off in production.

(seed → store by student_id → inject into system prompt → fresh thread

58 of 102

58

What we test on: tasks from real students

single-turn

60 questions, asked once

measures:

- retrieval (auto graded)

- right facts (key points)

- right kind of response (teach / redirect)

ex: “repo only has lessons

4,5,6,8,10, is that expected?”

sessions (multi-turns)

11 to 13 turns (more turns in part 3)

measures:

- in-session memory

under compaction

ex: plant “I want to learn about RAG”

at turn 0, probe it at turn 11

→ let's walk one session probe start to finish.

What we test on, three tasks each with its own dataset.

The four measures:

single-turn measures right facts (key points) + right kind of response (behavior: teach / redirect-to-support / acknowledge-feedback);
sessions measure memory probes under compaction;
personas measure profile personalization;

Composition (verified):
single-turn 60 = 37 answer_from_corpus + 10 redirect_to_support + 7 answer_general + 6 acknowledge_feedback.
Built from 151 real academy posts, kept 60 after review (cut 32, mostly stale).
Code/auto graded:�Retrieval (retrieval_metrics)

Citations (citation_metrics)�Behavior proxy (behavior_heuristic)

Telemetry

LLM judge�Key-point coverage “does the answer convey or clearly imply this atomic claim?”
Behavior = "right kind of response"

�

Sessions: 3 used (s01=13, s02=11, s08=12 turns = 36 turns/arm/trial), 4 graded probes each, so 12 probes/arm in the broad screen (1 trial) and 24 in the deeper bake-off (2 trials).
planted_facts — synthetic student facts we invent
The middle part, the filler/context turns are drawn from the same real-post pool the single-turn 60 came from.
Probe is the question from the student, that needs the planted fact
The pass/fail rule to the judge: "Must pick RAG evaluation and name hit rate + MRR; any other topic or metrics = fail."
Probe types:
fact_recall = the answer must use a fact the student stated many turns earlier (e.g. "RAG eval, hit rate, MRR").
preference_compliance = the tutor must honor a stated preference (OS/tooling/language/style, e.g. conda not venv). So "fact_recall 83% vs prod 33%" means: of the fact_recall probes (n=6 in that rerun), profile_memory passed 83%, prod 33%.
Every probe is judge graded

Personas: 10 profiles x 4 questions = 40.

GRADING - auto (code) vs validated LLM judge (blinded to the arm).
Rule: code computes what it reliably can; the judge handles nuance. Judge validated at 98% agreement / TPR 100% / TNR 96% vs human on 96 probes.

single-turn: AUTO = retrieval recall, citation match, behavior proxy, telemetry. JUDGE = key-point coverage (right facts), behavior (right kind of response), holistic, faithfulness.

sessions: AUTO = telemetry (cost/latency/tokens) + the gate (check_triggers: did compaction fire before the probe). JUDGE = the probe verdict (did it recall the planted fact) + holistic/faithfulness.

personas: AUTO = regex_any checks + anti-patterns (auto-fail). JUDGE = only the nuanced 'llm' checks. auto_pass = fail if an anti-pattern hits, else pass if all regex pass, else send to the judge.
PERSONAS in plain terms: each persona is a student profile (e.g. uses conda) seeded into long-term memory (here its actually just the inmemory setup); we ask 4 questions on a FRESH thread so only the stored profile carries the info, then check the answer reflects it - regex must-haves (e.g. 'conda'), anti-patterns it must avoid (e.g. 'venv'), plus optional llm checks for nuance. Measures long-term profile memory; off in production.

(seed → store by student_id → inject into system prompt → fresh thread

59 of 102

59

One session, start to finish

Turn 0 (plant)

“I'm a Unity dev. My weak topic is RAG evaluation. Dialogue must stream within 300 ms.”

Turns 1 to 10 (grow)

real course questions push the chat past the 30k trigger, so compaction fires.

Turn 11 (probe)

“One free evening, which topic do I drill and name two metrics?”

Grade

expected: RAG evaluation, hit rate, MRR.

Gate

check_triggers confirms compaction fired before turn 11; if it had not, the run is rejected (the probe would not test memory).

→ this is how session memory accuracy is produced, one graded probe at a time.

One worked example, and note this is the SESSIONS task specifically (a probe = a graded turn inside a session; single-turn and personas are graded differently).

Plant a fact at turn 0, grow the chat past the 30k trigger so compaction fires, probe the planted fact at turn 11.

Grade against expected facts (RAG evaluation; hit rate, MRR).
The gate (check_triggers) confirms compaction actually fired before the probe, else the probe wouldn't be testing memory and the run is rejected.

This trace is the 92% vs 38% on the next slides.
GRADING here: the probe verdict (did the answer recall the planted fact) is graded by the validated LLM judge, blinded to the arm - NOT by code. Code does the GATE only (check_triggers: did compaction fire before turn 11, else reject the run). So gate = automatic; probe pass/fail = judge (human-confirmed on all 96 probes).

60 of 102

60

The harness: run → grade → gate → report

run_battery (task) function drives the production agent code.

61 of 102

61

The harness: run → grade → gate → report

run_battery (task) function drives the production agent code.

run_battery

drives the real agent

1 bundle / turn

only step that costs $

62 of 102

62

The harness: run → grade → gate → report

run_battery (task) function drives the production agent code.

run_battery

drives the real agent

1 bundle / turn

only step that costs $

grade

code checks (free, offline), LLM

judge (subscription)

63 of 102

63

The harness: run → grade → gate → report

run_battery (task) function drives the production agent code.

run_battery

drives the real agent

1 bundle / turn

only step that costs $

grade

code checks (free, offline), LLM

judge (subscription)

check_triggers

THE GATE: did compaction

fire before the probe?

if not, reject the run

NO → reject the run

64 of 102

64

The harness: run → grade → gate → report

run_battery (task) function drives the production agent code.

run_battery

drives the real agent

1 bundle / turn

only step that costs $

grade

code checks (free, offline), LLM

judge (subscription)

check_triggers

THE GATE: did compaction

fire before the probe?

if not, reject the run

report

side-by-side tables

+ token-by-turn

curves

NO → reject the run

The judge sees only question, answer, and the grading criterion, so it cannot favor a strategy, and it matched human grades 98% on 96 probes.

→ we evaluate once, grade when we want afterwards.

65 of 102

65

What we ran: 11 presets, one variable

2 reference points: full_history (keep everything in context) and production (the live default).
6 one-change variants: sliding-window, prompt-compression, selective-retention, context-reset, in-context-history-retrieval, profile_memory
everything else held fixed: model, prompt, retrieval, sources, dataset. 660 turns, 0 API errors.
~590$ across the whole gemini study (all the batteries)

→ So which arm actually won?

The slide now shows the 11-arm screen only, one variable at a time.
2 reference points (full_history, production) + 9 one-change variants = 11 arms. The other 2 arms (aggressive, profile_memory) ran in the separate 4-arm bake-off and appear in the results/backup; 13 arms total across both runs.

Note: kb_off and retrieval_budget_30k are prod + a CLI flag, not standalone presets. 'Stratified single-turn' means the screening subset of 24 was sampled to preserve the mix of the 60 questions' types, so it's a fair mini-version. Counts (verified by counting bundles): bake-off 1,088 turns + screen 660 = 1,748 turns, 0 errors. $486 = bake-off ($323) + screen ($163); ~$590 across the whole study. (Don't say 'Part B/Part C' aloud - reserve 'Part' for the talk's Part 1/2/3.)

�In deep bake off:

single-turn: 60 cases (full battery) — "cases run | 60 | 60 | 60 | 60", recall rows at n=120 = 60 × 2 trials. So yes, all 60.
personas: 40 cases (full battery, 10×4) — "cases run | 40", n=80 = 40 × 2 trials. Full.
sessions: 3 cases — "cases run | 3", n=24 = 3 sessions × 4 probes × 2 trials. Not all 32.

In screening:�- The "24 of 60 single-turn + 3 sessions" describes the screen (Part C, 11 arms, ~$163) — and the screen did not run personas at all (c_report has no personas section: only 24 single-turn + 3 sessions)

Broad screen (left panel, top→bottom by score)

full_history (100%) — summarization=False, context_editing=False. No compaction at all; every prior message stays verbatim. The quality/memory upper bound and token worst case.

prompt-compression (100%) — summarization=False, compress_prompt=True. You're correct: PromptCompressionMiddleware runs three regexes over every message — runs of spaces/tabs → one space, strip line-leading whitespace, 3+ blank lines → 2. No LLM, drops zero facts (so 100% memory) but barely saves tokens. The code calls it a placeholder, explicitly "NOT a faithful LLMLingua-style compressor." prod's tool-output clearing stays on.

in-context-retrieval (83%) — incontext_history_retrieval: keep_recent=2, top_k=3, summarization off. Keeps the last 2 turn-blocks, embeds every older turn-block plus the current question (Cohere), and re-injects only the top-3 most similar old blocks. Retrieval replaces summarization.

production (58%) — prod, the live baseline. Summarizes older history once the prompt crosses 30k tokens (keeps last 20 messages) and clears old tool outputs at 5k tokens (keeps 5) — but clearing excludes retrieve_tutor_context, so the big retrieval payloads stay.

sliding-window (42%) — summarization=False, sliding_window_keep=12. Keep the last 12 messages, drop everything older, with the cut snapped to a user-turn boundary so no tool result is orphaned. Recency-only, no LLM.

selective-retention (25%) — prod's exact stack, but swaps in a tutoring-tuned summary prompt (SELECTIVE_RETENTION_SUMMARY_PROMPT) that explicitly tells the summarizer to preserve student facts / preferences / open thread. Only the summary prompt changes vs prod.

context-reset (17%) — prod stack with a "minimal state snapshot" prompt and summarize-early/keep-little knobs (trigger=15k tokens, keep=4 messages). An aggressive prefix rewrite that throws away history for a tiny snapshot.

It's prod's normal summarization stack, but with three changes: a custom summary prompt, an earlier trigger (summarization_trigger_tokens=15_000 vs prod's 30k), and far fewer kept messages (summarization_keep_messages=4 vs prod's 20). So it fires twice as eagerly and, when it does, collapses everything except the last 4 messages into a single "snapshot."

“””The conversation below will be discarded and replaced by your snapshot (a context reset). Write the minimal state needed to keep helping this student without re-asking: their level, goal, environment, and constraints (current values), the approach currently in progress, and the immediate question. Be brief; omit everything else. Respond ONLY with the snapshot.”””�

Deeper bake-off (right panel)

full_history (92%) — same arm as above.
aggressive (42%) — same prod-style stack, but every knob fires early and keeps little: summarize at 8k tokens (vs 30k) keeping 8 messages (vs 20); clear tool outputs at 2k tokens (vs 5k) keeping 2 (vs 5). The "how bad can cheap get" arm — it re-summarizes and re-retrieves mid-turn, so it's slow, pricey, and worst on memory.
production (38%) — same prod as above.

66 of 102

66

Broad screen · 8 presets · 1 trial (3 sessions) ~$88.12

full_history

100%

prompt-compression

100%

in-context-retrieval

83%

production

58%

sliding-window

42%

selective-retention

25%

context-reset

17%

Deeper exp · 3 presets · 2 trials (3 sessions) ~$62.17

full_history

92%

aggressive

42%

production

38%

What won: session memory recall, every method tested

profile_memory

75%

Session memory recall, every method we tested, shown as TWO panels because they are two different runs and are not directly comparable. Results are overall probe accuracy — all five probe types pooled

Left = Broad screen: 11 arms, 1 trial, n=12/arm (3 sessions x 4 probes).

Right = Deeper bake-off: the earlier 4-arm run, 2 trials, n=24/arm.
All bars match the run reports exactly.
The two anchors (full_history dark, production mid-grey) appear in BOTH panels and disagree (full_history 100 vs 92, production 58 vs 38): same 3 sessions and model, just 1 vs 2 trials and small n, which is the whole reason we read rankings not decimals and never compare across runs.

profile_memory and aggressive only ever ran in the bake-off; the other 9 only in the screen.
KB-off and 30k-budget are prod + a CLI flag, not standalone presets (the asterisk).

Caveat to say aloud: prompt-compression hitting 100% is real but it is a placeholder that only collapses whitespace, so it drops no facts yet barely saves tokens; do not read it as free perfect memory.

Selective retention has a tuned summarization prompt.
app/memory_presets.py
SELECTIVE_RETENTION_SUMMARY_PROMPT

Profile_memory extracts facts (tuned prompt)
You maintain a short profile of a student for an AI tutor.
Current profile (may be empty): {profile}
Latest exchange: Student: {query} / Tutor: {answer}
Rewrite the profile in at most 5 short lines. Keep only durable facts useful for future tutoring: skill level, goals, preferred language/tools, weak topics, current course/lesson. Drop one-off details. Return only the profile text; return NONE if nothing durable is known yet.

Why production is 38% (bake-off) vs 58% (screen)
The honest answer is that the gap isn't real signal — it's noise from tiny n.
That's exactly why the slide says "read the ranking, not the decimals, and never compare across runs."

* prod + a CLI flag. prompt-compression only collapses whitespace, so it keeps every fact but barely saves tokens.

Same 3 sessions and model, but 2 trials here vs 1 in the screen, and the two shared arms disagree: full_history 92 vs 100, production 38 vs 58. Small n, so read the ranking, not the decimals, and never compare across runs.

Broad screen (left panel, top→bottom by score)

full_history (100%) — summarization=False, context_editing=False. No compaction at all; every prior message stays verbatim. The quality/memory upper bound and token worst case.

prompt-compression (100%) — summarization=False, compress_prompt=True. You're correct: PromptCompressionMiddleware runs three regexes over every message — runs of spaces/tabs → one space, strip line-leading whitespace, 3+ blank lines → 2. No LLM, drops zero facts (so 100% memory) but barely saves tokens. The code calls it a placeholder, explicitly "NOT a faithful LLMLingua-style compressor." prod's tool-output clearing stays on.

in-context-retrieval (83%) — incontext_history_retrieval: keep_recent=2, top_k=3, summarization off. Keeps the last 2 turn-blocks, embeds every older turn-block plus the current question (Cohere), and re-injects only the top-3 most similar old blocks. Retrieval replaces summarization.

production (58%) — prod, the live baseline. Summarizes older history once the prompt crosses 30k tokens (keeps last 20 messages) and clears old tool outputs at 5k tokens (keeps 5) — but clearing excludes retrieve_tutor_context, so the big retrieval payloads stay.

sliding-window (42%) — summarization=False, sliding_window_keep=12. Keep the last 12 messages, drop everything older, with the cut snapped to a user-turn boundary so no tool result is orphaned. Recency-only, no LLM.

selective-retention (25%) — prod's exact stack, but swaps in a tutoring-tuned summary prompt (SELECTIVE_RETENTION_SUMMARY_PROMPT) that explicitly tells the summarizer to preserve student facts / preferences / open thread. Only the summary prompt changes vs prod.

context-reset (17%) — prod stack with a "minimal state snapshot" prompt and summarize-early/keep-little knobs (trigger=15k tokens, keep=4 messages). An aggressive prefix rewrite that throws away history for a tiny snapshot.

It's prod's normal summarization stack, but with three changes: a custom summary prompt, an earlier trigger (summarization_trigger_tokens=15_000 vs prod's 30k), and far fewer kept messages (summarization_keep_messages=4 vs prod's 20). So it fires twice as eagerly and, when it does, collapses everything except the last 4 messages into a single "snapshot."

The conversation below will be discarded and replaced by your snapshot (a context reset). Write the minimal state needed to keep helping this student without re-asking: their level, goal, environment, and constraints (current values), the approach currently in progress, and the immediate question. Be brief; omit everything else. Respond ONLY with the snapshot.�

Deeper bake-off (right panel)

full_history (92%) — same arm as above.
aggressive (42%) — same prod-style stack, but every knob fires early and keeps little: summarize at 8k tokens (vs 30k) keeping 8 messages (vs 20); clear tool outputs at 2k tokens (vs 5k) keeping 2 (vs 5). The "how bad can cheap get" arm — it re-summarizes and re-retrieves mid-turn, so it's slow, pricey, and worst on memory.
production (38%) — same prod as above.

67 of 102

67

Broad screen · 8 presets · 1 trial (3 sessions) ~$88.12

full_history

100%

prompt-compression

100%

in-context-retrieval

83%

production

58%

sliding-window

42%

selective-retention

25%

context-reset

17%

Deeper exp · 3 presets · 2 trials (3 sessions) ~$62.17

full_history

92%

aggressive

42%

production

38%

→ keeping everything wins memory in both runs. Now the cost side.

What won: session memory recall, every method tested

profile_memory

75%

Session memory recall, every method we tested, shown as TWO panels because they are two different runs and are not directly comparable. Results are overall probe accuracy — all five probe types pooled

Left = Broad screen: 11 arms, 1 trial, n=12/arm (3 sessions x 4 probes).

Right = Deeper bake-off: the earlier 4-arm run, 2 trials, n=24/arm.
All bars match the run reports exactly.
The two anchors (full_history dark, production mid-grey) appear in BOTH panels and disagree (full_history 100 vs 92, production 58 vs 38): same 3 sessions and model, just 1 vs 2 trials and small n, which is the whole reason we read rankings not decimals and never compare across runs.

profile_memory and aggressive only ever ran in the bake-off; the other 9 only in the screen.
KB-off and 30k-budget are prod + a CLI flag, not standalone presets (the asterisk).

Caveat to say aloud: prompt-compression hitting 100% is real but it is a placeholder that only collapses whitespace, so it drops no facts yet barely saves tokens; do not read it as free perfect memory.

Selective retention has a tuned summarization prompt.
app/memory_presets.py
SELECTIVE_RETENTION_SUMMARY_PROMPT

Profile_memory extracts facts (tuned prompt)
You maintain a short profile of a student for an AI tutor.
Current profile (may be empty): {profile}
Latest exchange: Student: {query} / Tutor: {answer}
Rewrite the profile in at most 5 short lines. Keep only durable facts useful for future tutoring: skill level, goals, preferred language/tools, weak topics, current course/lesson. Drop one-off details. Return only the profile text; return NONE if nothing durable is known yet.

Why production is 38% (bake-off) vs 58% (screen)
The honest answer is that the gap isn't real signal — it's noise from tiny n.
That's exactly why the slide says "read the ranking, not the decimals, and never compare across runs."

* prod + a CLI flag. prompt-compression only collapses whitespace, so it keeps every fact but barely saves tokens.

Same 3 sessions and model, but 2 trials here vs 1 in the screen, and the two shared arms disagree: full_history 92 vs 100, production 38 vs 58. Small n, so read the ranking, not the decimals, and never compare across runs.

Broad screen (left panel, top→bottom by score)

full_history (100%) — summarization=False, context_editing=False. No compaction at all; every prior message stays verbatim. The quality/memory upper bound and token worst case.

prompt-compression (100%) — summarization=False, compress_prompt=True. You're correct: PromptCompressionMiddleware runs three regexes over every message — runs of spaces/tabs → one space, strip line-leading whitespace, 3+ blank lines → 2. No LLM, drops zero facts (so 100% memory) but barely saves tokens. The code calls it a placeholder, explicitly "NOT a faithful LLMLingua-style compressor." prod's tool-output clearing stays on.

in-context-retrieval (83%) — incontext_history_retrieval: keep_recent=2, top_k=3, summarization off. Keeps the last 2 turn-blocks, embeds every older turn-block plus the current question (Cohere), and re-injects only the top-3 most similar old blocks. Retrieval replaces summarization.

production (58%) — prod, the live baseline. Summarizes older history once the prompt crosses 30k tokens (keeps last 20 messages) and clears old tool outputs at 5k tokens (keeps 5) — but clearing excludes retrieve_tutor_context, so the big retrieval payloads stay.

sliding-window (42%) — summarization=False, sliding_window_keep=12. Keep the last 12 messages, drop everything older, with the cut snapped to a user-turn boundary so no tool result is orphaned. Recency-only, no LLM.

selective-retention (25%) — prod's exact stack, but swaps in a tutoring-tuned summary prompt (SELECTIVE_RETENTION_SUMMARY_PROMPT) that explicitly tells the summarizer to preserve student facts / preferences / open thread. Only the summary prompt changes vs prod.

context-reset (17%) — prod stack with a "minimal state snapshot" prompt and summarize-early/keep-little knobs (trigger=15k tokens, keep=4 messages). An aggressive prefix rewrite that throws away history for a tiny snapshot.

It's prod's normal summarization stack, but with three changes: a custom summary prompt, an earlier trigger (summarization_trigger_tokens=15_000 vs prod's 30k), and far fewer kept messages (summarization_keep_messages=4 vs prod's 20). So it fires twice as eagerly and, when it does, collapses everything except the last 4 messages into a single "snapshot."

The conversation below will be discarded and replaced by your snapshot (a context reset). Write the minimal state needed to keep helping this student without re-asking: their level, goal, environment, and constraints (current values), the approach currently in progress, and the immediate question. Be brief; omit everything else. Respond ONLY with the snapshot.�

Deeper bake-off (right panel)

full_history (92%) — same arm as above.
aggressive (42%) — same prod-style stack, but every knob fires early and keeps little: summarize at 8k tokens (vs 30k) keeping 8 messages (vs 20); clear tool outputs at 2k tokens (vs 5k) keeping 2 (vs 5). The "how bad can cheap get" arm — it re-summarizes and re-retrieves mid-turn, so it's slow, pricey, and worst on memory.
production (38%) — same prod as above.

68 of 102

68

What “production” actually costs and scores

	single-turn	sessions
est. cost / turn	~$0.36	~$0.24
input tokens / turn	~216k	~142k
median first-token	~33s	~21s
quality	92% behavior · 68% key-pts	38% memory probes

This is the production anchor: the production preset on gemini-3.5-flash. The model is held constant; the preset is what varies. (Other arms: previous chart.)

→ and if we compare to full_history?

69 of 102

69

The surprise: compaction didn't pay

sessions, 11 to 13 turns	full_history	production
cost / turn	$0.11	$0.24
first-token	17s	21s
memory recall	92%	38%

keep-everything wins all three.

70 of 102

70

The surprise: compaction didn't pay

sessions, 11 to 13 turns	full_history	production
cost / turn	$0.11	$0.24
first-token	17s	21s
memory recall	92%	38%

keep-everything wins all three.

why production loses

1) summarizing rewrites the cached prefix, a cache miss: production sends ~42% fewer tokens than full_history yet pays ~2× more (full_history bills ~87% of input at the 4× cache discount).

2) cleared tool-outputs get re-retrieved: pay, drop, pay again.

71 of 102

71

The surprise: compaction didn't pay

sessions, 11 to 13 turns	full_history	production
cost / turn	$0.11	$0.24
first-token	17s	21s
memory recall	92%	38%

keep-everything wins all three.

why production loses

1) summarizing rewrites the cached prefix, a cache miss: production sends ~42% fewer tokens than full_history yet pays ~2× more (full_history bills ~87% of input at the 4× cache discount).

2) cleared tool-outputs get re-retrieved: pay, drop, pay again.

But these are the results in these conditions, with 11 to 13-turn sessions → we can’t justify compaction.

→ That sets up Part 3 where we test under different conditions

72 of 102

72

PART 3

So when does compaction actually matter?

cached chat → documents + tools → local & at scale

73 of 102

73

Should you ever compact?

On Gemini 3.5 Flash, keep-everything won
But full history on a frontier model is expensive.
So when is compaction actually worth it?

every number here is from our own eval runs on the tutor's corpus.

74 of 102

74

First, what fills the context?

Long chat history → the one detail that matters gets buried
Pasted document → too big to fit the window
Big tool output → pages of useless verbatim logs
Each type needs a different fix.

75 of 102

75

First lever: a cheaper model

Same test, swap the model: Gemini 3.5 Flash → DeepSeek V4 Flash.

~18x cheaper per turn, and keep-all still wins.

	Gemini 3.5 Flash	DeepSeek V4 Flash
cost / turn	~$0.11	~$0.006
cache discount	~10x	~50x
keep-all wins?	yes	yes

76 of 102

76

But does it still remember?

Keep-all 95% vs summarize 32%, summarizing drops the detail a later question needs.

here's how I tested memory, I took the conversation where the student mention little details throughout — things like their setup, the error they were seeing, or what they'd already tried. Then later in the chat, I asked a question that depended on one of those details, and checked whether the model could still answer correctly. With the full conversation, it got the answer right 95% of the time. When I summarized the conversation first, that dropped to just 32%. And the reason is pretty simple. As soon as you summarize a conversation, you're rewriting it into a shorter version, and that shorter version has to leave things out. So if a later question depends on one of those missing details, the information is just gone. If you keep the entire conversation, though, the original detail is still there — the model can go back and find it. So summarizing remembered less. And the sample here is small, twenty-two probes, but the gap is big enough that the direction is clear. So keeping everything clearly wins on memory. But it also means re-sending the whole conversation every single turn, which sounds expensive. So does it actually cost the most?

77 of 102

77

The cheapest run sends the most tokens

Keep-all bills the most tokens (296k/turn) yet is the cheapest setup, 97% cached. At 36 turns: 1.78M tokens.

So on DeepSeek, the setup that sends the most tokens is actually the cheapest to run. Keeping the full history bills the most tokens per turn, around 296 thousand, but it still comes out cheapest, because about 97% of those tokens are cached, and cached tokens are very cheap.

Summarizing works the other way: every turn it makes the model read and write new tokens. I ran this out to a 36-turn conversation, about 1.78 million tokens, and keeping everything still came out ahead. And it's not just DeepSeek, we saw the same on Gemini: the summarizing setup used about 42% fewer tokens but cost roughly twice as much per turn, because it kept breaking the cache. So the cheaper model is what brought the cost down in the first place, but the reason keeping everything wins over summarizing isn't the model, it's the caching, and that's why we saw it on both. The next thing I wanted to check was whether that holds when the conversation gets really long.

78 of 102

78

Does it hold at long context?

Single-fact recall held to 800k (1M is a hard wall). Our job is single-fact retrieval, not long-history reasoning.

79 of 102

79

At scale, compaction is economics

Cheap per turn isn't cheap at scale: $18k–180k/mo at 100k–1M turns/day, even on DeepSeek.

80 of 102

80

Going local: can we just cache?

At scale, the bill pushes us to run our own model instead of paying per token.
But our own hardware means a small model, only a ~32k context window.
The cloud's move was keep-everything plus caching. Can we do that locally?

81 of 102

81

We can't: it doesn't fit

No, the context doesn't fit in 32k.
Our lessons are already bigger than that, and students paste long logs that blow past it.
And if it doesn't fit, there's nothing to cache, so locally we have to compress or retrieve.

82 of 102

82

Locally, keep-all stops winning

Forced to compact, every method lands in a 27–40% band, a 4–5x bigger 32B stays in it. The 32k window is the limit, not the model.

83 of 102

83

Local docs: retrieve, don't stuff

Stuffing one long doc overflows the window at every size (a 1-token stub). RAG wins everywhere, ~25–65s vs ~340s, ~5x throughput.

84 of 102

84

Retrieve, and make it hybrid

We can't send the full context, so we have to retrieve. Dense-only collapsed at 400k (buried fact 0%); keyword/BM25 held 100%. The tutor keeps a keyword path.

The default is semantic search, matching on what the text means, and that's the dense RAG you see on the chart. It mostly works until the fact you need is buried far enough in. At 400k, that dense search missed the buried fact entirely, zero percent, while plain keyword search, BM25, still got it 100% of the time. So semantic search on its own isn't enough, you want both, and that's why our tutor's retrieval is hybrid: it runs a keyword search alongside the semantic one. So locally, the answer is to retrieve, and to make it hybrid. To be clear about that zero, that's the fact buried right in the middle, the hardest spot. Dense still found facts near the end, so its overall recall at 400k was about half. It's the buried-middle case where dense drops off and keyword holds, and the BM25 100% is across the board. Which leaves one last question: how does all of this local setup compare to the cloud?

85 of 102

85

	Local 32B	DeepSeek V4 Flash	Gemini 3.5
Cost / turn	~$0 (own GPU)	~$0.006 ✓	~$0.11 ✓
Chat memory (keep-all)	33% †	95% †	92% †
Doc Q&A (RAG)	100% ✓	n/a ‡	n/a ‡
Speed (first token)	~20–350s ✓	~1–20s ✓	~17–76s ✓
Throughput	~1,300 turns/day/GPU *	elastic (cloud)	elastic (cloud)

Local doesn't match cloud on chat (window-capped), but it kills the per-token bill, and RAG carries docs.

✓ from our runs † within-model, directional (not a matched test) ‡ not tested (RAG was local-only) * estimate, source not in repo

Does local match the cloud?

86 of 102

86

Keep-all 95% vs summarize 32%

share of earlier chat facts still recalled. Don't summarize your history away.

Finding a fact in a long context is easy

no rot for our job (checked to ~800k).

The cheapest run sends the most tokens

caching makes the repeated context nearly free.

At ~10k turns/day	$ / month
Gemini 3.5	~$34k
DeepSeek V4 Flash	~$1.9k
Local SLM	~$0 / token*

Scenario: ~10k turns/day (≈ a few thousand students).

Gemini to DeepSeek is the big lever (~18x). Local is about privacy and throughput, not cost at this scale.

*~3 Macs, throughput-bound.

Don't compact by default. Name the constraint first: window, cost, or throughput.

What we found, in our case

So, for our AI tutor: on memory, keeping the whole chat recalled about 95% of the details that came up earlier, versus 32% if you summarize first, so it's better not to summarize your history away. ��On long context, finding a single fact is easy for the model, no real rot for our job, we checked it out to about 800k. ��On cost per turn, the cheapest run is actually the one sending the most tokens, because caching makes re-sending the same context nearly free. And on cost at scale, at a realistic level for us, a few thousand students, call it about ten thousand turns a day: Gemini is around 40 thousand a month, DeepSeek about nineteen hundred, and local is basically zero per token, just about three Macs, though it's throughput-bound. Going local saves on cost a bit more, but at this scale it's really about privacy and control, not cost. So the main thing to take away: don't compact by default, name the constraint first, whether that's the window, the cost, or throughput.

87 of 102

87

So what should you actually run?

Model: DeepSeek V4-Flash, a cached cloud model. Caching makes keep-all the cheapest.

Retrieval: hybrid, dense + BM25 + rerank.

Memory: keep everything; compaction is an off-by-default fallback (a ~30k-history knob). The only hard stop is the ~1M window.

~$0.006 / turn · first reply ~1–2s typical · 95% recall

That's the exact tutor, the retrieval, the agent, and the evals, that we walk you through building in the course.

So this is where we landed for our tutor. The model we went with is DeepSeek V4-Flash, a cloud model, and since it caches, the cheapest thing we can do is just keep everything. On top of the model, we run hybrid retrieval, so we catch both the semantic matches and the exact buried facts. For memory we keep the whole conversation, and compaction stays off by default. It's just a fallback knob sitting around 30k tokens of history, and the only real hard stop is the model's window, about a million tokens. If you're just getting started, any cached cloud model with keep-all will take you a long way. For us, the best setup was DeepSeek with hybrid retrieval and keep-all, roughly 18 times cheaper than Gemini. In practice, that's about six tenths of a cent per turn, a first reply in a second or two, and ninety-five percent recall. So that's the setup, the tutor, the retrieval, the agent, and the evals behind it.

88 of 102

88

Want to build one yourself?

These are a few of the experiments. We ran many more.

Want to go deeper? Build this exact tutor, evals and all.

Full Stack AI Engineering Course, academy.towardsai.net

Questions?

89 of 102

89

PART 3

So when does compaction actually matter?

cached chat · document/tool context · local and at scale

90 of 102

90

Should you ever compact?

We saw keep-everything win on Gemini, and it cost us $600
So: can a cheaper or local model match it?
The real question isn't whether to compact, it's : the window, the cost, or the rot.

every number here is from our own eval runs on the tutor's corpus.

91 of 102

91

First, what kind of context is it?

Context isn't one thing.

A chat history, a document, and a tool output each break for different reasons.

A chat history grows every turn; a document or a big tool output can fill the window in one shot.
The right method depends on the shape, identify it first.

92 of 102

92

Cheaper model, same answer

Same verdict, keep-all wins, at about 17x lower cost (same sessions test as Gemini). Quality judged within each model, not head-to-head.

	Gemini 3.5	DeepSeek V4
cost / turn	~$0.11	~$0.006
cache discount	~10x	~50x
keep-all wins?	yes	yes

93 of 102

93

The cheapest run sends the most tokens

Keep-all bills the most tokens (296k/turn) yet is the cheapest arm. At 36 turns, even more extreme: 1.78M tokens, 97% cached.

94 of 102

94

But does it still remember?

Keep-all 95% vs summarize 32% (DeepSeek sessions, n=22).

Cost is one thing, but does the cheaper model still remember? So I tested memory the same way. I had the student mention little details throughout the conversation, their setup, the error they were seeing, what they'd already tried. Then later I asked a question that depended on one of those details, and checked whether it could still answer. With the full conversation, it got the answer right ninety-five percent of the time. When I summarized the conversation first, that dropped to thirty-two percent. And the reason is pretty simple. Summarizing rewrites the chat into a shorter version, and that shorter version has to leave things out. So if a later question depends on one of those missing details, it's just gone. If you keep the whole conversation, the detail is still there, and the model can go back and find it. (If anyone asks how these were scored: an LLM judge we validated at about ninety-eight percent agreement with human grades, and the Gemini version of this same result was human-confirmed.)

95 of 102

95

Go local, and keep-everything stops winning

On the cloud keep-everything won; here it's just one of the pack (27-40% band). The window is constraint, not the model, and a 4-5x bigger model doesn't move it.

So here's the part that flips. On the cloud, keeping everything won, it was the cheapest and it remembered the most. The moment you go local, that flips. Keeping everything stops winning. Actually nothing really wins, every method lands in the same low band, around thirty to forty percent, and keep-everything is right in the middle, no better than the rest. And the reason is the window. It's small, thirty-two thousand tokens, so a long conversation just doesn't fit. Now you might be thinking, just use a bigger local model. We tested that, a thirty-two billion one, four to five times bigger, and it landed in exactly the same band, because a bigger model doesn't give you a bigger window. And to be clear, this isn't about caching either, local models cache too. It's the window. Once the conversation is bigger than the window, the old turns fall off the back no matter what you do.

96 of 102

96

Local docs: retrieve, don't stuff

One long lesson stuffed in one turn: it overflows the window on every size (a 1-token stub). RAG wins at every size, and it's far faster: ~25-65s vs ~340s.

97 of 102

97

Context rot? For our job, it held

Single-fact recall held to 800k. Reasoning across a long history is a coding-agent job, not ours, by design.

So the big fear with long context is rot. The idea that even when everything fits, the model quietly misses a fact buried deep inside. So we tested that directly. We hid a fact in a long context, turned retrieval off so the model had to read for it, and asked for it back at different lengths. And for the kind of question our tutor actually gets, pulling one fact, it held up. It stayed high all the way out to eight hundred thousand tokens, both in a document and in a conversation. Now you might ask, what about harder stuff, reasoning across a whole long history instead of finding one fact. And that does get harder, it's a real thing. But this was a decision we made on purpose. Our tutor answers one focused question at a time, it finds the right piece and answers it. So single-fact retrieval isn't a shortcut, it's exactly the right test for what we built. We're not saying context rot is solved. We're saying it isn't our problem, because it isn't our job. One honest note, if it comes up. We ran ten samples per length up to four hundred K, and five at six hundred and eight hundred K, because those longest runs get expensive and we were being cost-conservative. So treat the two longest points as directional, the pattern is what matters, not the exact number.

98 of 102

98

If you retrieve, use hybrid

Dense retrieval collapsed at 400k; keyword (BM25) held 100%. The tutor's hybrid keeps a keyword path.

Okay, so if a buried fact is the problem, the obvious fix is to retrieve it instead of stuffing everything. So I tested the retrievers head to head on the cases that were failing. The dense one, embeddings plus a reranker, held up at moderate lengths but gave out at four hundred K. It just couldn't surface a made-up codename among hundreds of chunks. Plain keyword search, BM25, found it every single time. And to be precise about that collapse, the zero percent is the fact buried right in the middle, the hardest spot. Dense still found facts near the end, so its overall recall at four hundred K was about fifty percent. I'm showing the buried-middle case on purpose, because that's the hard one, and that's where dense gives out while keyword holds. So the lesson isn't retrieve versus don't, it's which retriever. You want a keyword path, not embeddings alone, and our tutor's hybrid already has one.

99 of 102

99

At scale, compaction is economics

Cheap per turn isn't cheap at scale: $18k-180k/mo. RAG ~5x throughput.

So far the reasons to keep everything were quality and convenience. But one thing does force your hand, cost at volume. The bill is mostly the retrieval payloads, not the chat. Per turn these numbers look tiny, a cent or two, basically free on local. But multiply by volume. A free tool that goes a little viral, a few hundred thousand to a million turns a day, is eighteen to a hundred and eighty thousand dollars a month, even on the cheap model. And locally there's throughput, stuffing a whole document is about five times slower per turn than retrieving the piece you need. So at scale, you compact, but for the cost, and retrieval is the tool.
If anyone asks how I priced this: both lines are the same workload, our normal sessions of about a dozen turns, one run on Gemini and one on DeepSeek, so it is a fair head to head. And it is actually a little generous to the expensive model. The longer a chat runs, the more history rides on every single turn, so Gemini's cost per turn keeps climbing as the conversation grows. I used the shorter sessions number here, so the real gap at scale is wider than what is on the screen, not narrower.

100 of 102

100

Compaction is the exception, not the rule

If you can cache, the cheapest thing is to keep everything.

Okay, so what do you take away from all of this. We walked in assuming we'd need a whole toolbox of compaction tricks. We measured, every way we could, and we mostly didn't. When the model can cache, keeping everything is cheaper, faster, and it remembers more. So compaction isn't the default move, it's the exception. You reach for it only when something actually forces your hand. A window too small to hold the context. Real cost at scale. Or one buried fact you have to surface. So the thing to take with you is this. Don't compact just because the context got long. Name the kind of context you have, name what's actually biting, and most of the time the move is simply to keep everything. And the bigger lesson, the one I'd want you to remember, is that we stopped guessing and we measured, and measuring flipped a lot of the advice we walked in with.

101 of 102

101

What we found, in our case

1.78M

tokens in the cheapest run (97% cached)

~18×

cheaper per turn: DeepSeek vs Gemini, same result

95% vs 32%

memory: keep-all vs summarize-first

800k

single-fact recall held this far (1M = hard wall)

$18k–180k

API bill / month at scale → why you go local

100% vs 0%

buried fact at 400k: keyword (BM25) vs dense

In our case, easiest: if you can cache, keep everything. Cheapest at scale: go local and retrieve (hybrid).

So here's the quick scoreboard of what we found, and again, this is in our case, for our tutor. The cheapest run was the one sending the most tokens, 1.78 million of them, 97% cached. The cheap cloud model was more than ten times cheaper per turn than the frontier one, and gave the same result. On memory, keeping everything scored 95% versus 32% for summarizing. Single-fact recall held all the way out to 800k. At real volume, the bill is eighteen to a hundred and eighty thousand a month, and that's the thing that pushes you local. And once you retrieve, keyword search held 100% on a buried fact where dense search dropped to zero. So the one-line version: in our case, if you can cache, keep everything, and you only give that up when the window or the budget forces you. So the simplest setup is cache and keep everything; the one that scales is go local and lean on retrieval. And honestly, this is just a slice of what we looked at.

102 of 102

102

Want to build one yourself, evals and all?

Full Stack AI Engineering Course

• Build real products with LLMs: context engineering, RAG, agents, evals.

• The exact stack behind this tutor, end to end.

• 60 hours, hands-on. Build and evaluate your own.

academy.towardsai.net