FlakeGuard
An agent that reads your CI history and decides which failures mean something.
Why now
“[unsupervised AI] fix: propagate
task annotations during worker execution”
dask/distributed · PR #9340 · one file changed: worker.py
3 pre-existing tests in 3 unrelated modules failed
17 of 17 cells each · nobody had touched those test files
The finding
A test matrix is a failure multiplier
94.4%
of individual matrix-cell runs
pass (6,119 of 6,479)
162 of 191
whole scheduled runs
go red
34 cells, each flipping its own coin. The run fails when any one lands wrong.
The same data, two questions
How often does one cell pass? · How often does the whole run pass?
29 of 191 runs were green. A 94.4% cell, thirty-four times over, is a 15% run.
Six verdicts
masked flake
fails, then passes
regression
fails everywhere
chronic
flaky for months
platform-specific
one OS, one cell
environment break
started on a date
unclear
not decidable yet
“unclear” is a correct answer, not a failure.
“environment break” has no instance in this data. We implemented it and did not fabricate one.
The determinism boundary
The model never carries a number.
Every statistic is computed in Python and pinned into the prompt. Numbers are quoted, never calculated.
The model never decides to write.
Classification is a judgement. Modifying a repository is a plain Python function a prompt cannot reach.
We added one sentence to one verdict definition.
Temperature 0. Same model, same evidence. Then we re-ran every fixture, ten times.
The case we targeted
0/10 → 10/10
An unrelated verdict, moved across the action threshold
0/10 → 4/10
Fabricated numbers across 80 outputs
0 → 9
We reverted it and kept the worse score.
Architecture
simplified for legibility — full diagram in docs/architecture.svg
deterministic Python
LLM call
the gate
GitHub
Actions API
parse JUnit
per cell
SQLite
704,323 rows
stats.py
Wilson · dispersion
cheap checks
no model call
classifier
correlation
ACTION GATE
confidence ≥ threshold
drafter
issue · PR
never merged
review queue · deferred · nothing written
explain — read-only by construction
Every orange box sits inside a blue path. A model never touches the API, the database, or the decision to write.