1 of 14

Designing an Onion:

Domain-Scoped AI Context at Monorepo Scale

Na'aman Hirschfeld

Principal Systems Engineer @ ServiceNow (Armis)

github.com/Goldziher

linkedin.com/in/nhirschfeld

2 of 14

WHERE WE STARTED

One Bazel monorepo, many languages, several products
1,000+ service folders side by side; one product spans 80+ of them
No folder you can point at and say “this is Product A”

services/

libs/

frontends/

deploy/

data/

tools/

Product A's code

every other product's code

The question

Product B dev: “show me this customer's details”

Root skill fires

written when the repo held only Product A

Wrong system

queries Product A's system and data

Plausible answer

wrong product, no error, unseen in review

• We work in one large Bazel monorepo, in many languages, and it serves several products. The code isn't namespaced by product. One services folder holds over a thousand service folders side by side, and a single product spans more than eighty of them, plus its frontends and its deploy configuration. There's no folder you can point at and say: that's Product A.

• On top of that, words like customer, account or pipeline mean different things in different products. Knowledge that is right in one place is a wrong turn in another. Hold that failure in mind.

• So the question for this talk: how do you give a coding agent the right product knowledge in a repo like this, without it bleeding into everyone else's work? And how do you know it worked? That second part is where we spent most of our effort, and I'll show you the numbers, including how much weight each one can carry.

3 of 14

PRIOR ART

First try: a plugin marketplace per domain

marketplaces

130

plugins

enabled by default

Far from the code: knowledge sat in a tooling folder
Disabled means invisible: agents only see enabled plugins
So every engineer had a different subset, and a different repo

• Our first attempt follow the obvious one: Claude Code plugin marketplaces, one per domain, kept together in a tooling folder. On paper they're ideal, neat installable bundles of skills, agents and commands. When we counted, that was twenty-six marketplaces, a hundred and thirty plugins, and exactly one plugin enabled by default. Underneath: 237 skill files, 46 agents, 88 commands, and 60 CLAUDE.md files.

• Three things went wrong. The knowledge sat far from the code it described. Disabled means invisible: an agent can only discover skills in plugins that are enabled. And the plugin UI isn't built for twenty-six marketplaces, so every engineer ended up with a different subset.

• The same repo behaved differently for every person: same prompt, different answers, depending on whose settings you had.

4 of 14

THE GOAL

Maintain AI context as software

Versioned

Owned

Linted

Reviewed

Tested

Measured

Each session gets its domain's knowledge, and nothing else

• So we stepped back and asked a different question: what if we maintained AI context the way we maintain software? Software lives next to the code it serves. It has owners. It's linted, reviewed and tested, and it's measured before and after a change.

• That gives six words, and I'll use them as a thread for the rest of the talk. Versioned: context lives in the repo, beside the code it describes, so a fresh clone has every layer. Owned: each folder's layer belongs to the team that owns that code. Linted: a linter checks the config in CI on every change. Reviewed: changes go through pull-request review by those owners. Tested: real sessions run against hundreds of test cases. Measured: the final pass or fail comes from test cases that were never used while we tuned.

• And the goal all of it is: each session gets its domain knowledge, and nothing else.

5 of 14

THE DESIGN

The onion: each layer loads when work reaches it

Load behaviour probed with a unique sentinel token per file, Claude Code 2.1.306.

Repo root

every session start

Standards, a products map, shared skills, generic subagents, the skill index

Area folder

on first Read inside

Its CLAUDE.md, rules and skills, all at once

Service folder

on first Read inside

The same, one level deeper; the closest CLAUDE.md wins, the root still loads

The file

glob-matched rules

Only the rules whose glob matches this file

Grep and Glob load nothing

• Here's the onion. Each ring is a layer of the repository, and each one loads only when the work reaches it.

• The outer ring is the root, and every session pays for it. So it's generic and owned by one platform team: shared engineering standards, a products map that says which product lives at which paths, a short list of cross-cutting skills, generic subagents, and a local skill index I'll come back to. Nothing in it describes how a product works.

• The next ring is an area folder, like services or a frontend. The first time the agent reads or writes a file inside it, that folder's CLAUDE.md, its rules and all of its skills load. A service folder one level down works the same way, and the closest CLAUDE.md wins when instructions conflict.

• At the core is the file itself. Rules scoped by glob load when a matching file is read.

• We tested this empirically by putting a unique sentinel token in every file and asked sessions what they could see.

• Ownership follows the same rings: the platform team owns the root, and each folder's layer belongs to whoever owns that code.

6 of 14

WHERE THINGS LIVE

The repository tree, before and after

Before

repo/

├─ CLAUDE.md

product terms

├─ tools/

│ └─ marketplaces/

26 of them

│ ├─ team-a/plugins/

skills, agents

│ ├─ team-b/plugins/

commands

│ └─ …

├─ services/

│ ├─ a-api/

no context here

│ └─ b-ingest/

or here

└─ frontends/

~/.claude/settings.json

per engineer

After

repo/

├─ CLAUDE.md

standards + map

├─ .claude-plugin/

8 plugins

│ └─ marketplace.json

one marketplace

├─ .claude/

│ ├─ settings.json

declares it

│ ├─ rules/

glob-scoped

│ └─ skills/ agents/

shared, generic

└─ services/

├─ CLAUDE.md

area guide

└─ a-api/

├─ CLAUDE.md

service guide

└─ .claude/

rules, skills

• Before: the root CLAUDE.md carried product terms, and the domain knowledge lived in twenty-six marketplaces in a tooling folder, far from the services it described. What a session got depended on each engineer's personal settings. Nobody could see the whole picture, and nobody owned it.

• After: the root holds only generic content: the standards and the products map, glob-scoped rules, shared skills and generic agents, and one settings file.

• There's exactly one plugin marketplace, at the root, with eight plugins. The root settings declare it once, with auto-update on and no plugin enabled by default, so everyone sees the same thing. Each plugin lives next to the code it serves, and the marketplace lists them by relative path.

• Everything specialised sits beside its code: an area guide, a service guide, that service's rules and skills. Nothing to install, because a fresh clone has it all. And a guard test fails CI if someone adds a second marketplace or a stray settings file.

• Why does this matter for the failure I opened with? Because a product's skills now live in its own folder, owned by its team, and a root session sees none of them.

7 of 14

THE LAYERS

Nested layers load on first read

Load behaviour probed with a unique sentinel token per file, Claude Code 2.1.306.

First Read or Write in a folder loads its CLAUDE.md, rules and skills
Grep and Glob load nothing nested, so searching stays clean
6 generic subagents at the root; nested ones are never found
A subagent reading a product file gets that folder's layer too

Session start

root layer only

Grep / Glob

nothing nested loads

First Read in a/

the folder is touched

Layer loads

its CLAUDE.md, rules, skills

Subagent reads it

gets the same layer

Probe, don't trust the docs: a unique sentinel token in every file, one question per session

• Three elements do the layering: nested folders, rules and skills. Nesting first, because it's where we did the most probing. Any folder, a service, a frontend, a deploy area, can carry its own CLAUDE.md, rules and skills, at whatever depth its code lives.

• We tested exactly when they load, with the sentinel tokens I mentioned. The first Read or Write of a file in a folder loads that folder's CLAUDE.md, its rules and all of its skills. Grep and Glob load nothing nested, so an agent searching the repo doesn't drag every product into its context.

• Subagents are the interesting exception. Claude Code only discovers subagents at the root, never in nested folders, not even after a read. So we keep only six generic ones at the root: a backend engineer, a code reviewer, a test engineer and so on.

• And they still get specialised. When a subagent reads a product file, that folder's CLAUDE.md and skills load for it too. Specialisation by composition, instead of a copy of every agent per product.

• For ownership we leverage GitHub’s CODEOWNERS to give each folder to its owning team.

8 of 14

THE LAYERS

Rules, skills and what stays at the root

Rules

Always-on, or glob-scoped with paths in the frontmatter. Loaded automatically when a matching file is read.

Skills

Loaded on demand: the model matches the description, or you type /name. Only name and description sit in context.

Gated skills

paths: hidden and uncallable until a matching file is read. disable-model-invocation: slash command only.

Slash commands

A nested skill is unknown until its folder is read, so commands needed from the root live in the root skills.

Root: global skills, plus the skills that must be root slash commands. Everything product-specific: nested, next to its code

• Inside the layers, two kinds of files carry the knowledge: rules and skills. They load differently, and that decides where each one lives.

• Rules are either always-on or glob-scoped. A rule carries a paths list in its front matter and loads automatically when the agent reads a matching file. Python, TypeScript and Terraform standards, lock files, generated files: each shows up when the agent touches such a file. We can't point at a product folder in this monorepo, but we can point at file types and naming patterns. A root rule with no paths would load in every session, so we have none.

• Skills are loaded on demand. Only a skill's name and description sit in context, about eighty tokens each. The model loads one when its description matches the task, or you load it yourself by typing its slash command.

• A skill can also be gated. Give it a paths list and it stays hidden until the agent reads a matching file; before that it is not listed and cannot be invoked. Mark it disable-model-invocation and the model never sees it, so it is a slash command only, at about zero cost.

• Now the part that surprised us. Claude Code discovers nested skills and commands only after it has read into that folder. Before that, a nested slash command is simply unknown. So a command that must work from the root has to live in the root skills folder, guarded so the model never fires it on a loose word.

• That is the design. The root keeps only global skills, plus the few skills that must be root slash commands. Everything product-specific lives nested, next to its code.

9 of 14

DISCOVERY

Dynamic skill discovery

ai-rulez: open source, by the speaker — https://github.com/Goldziher/ai-rulez

The problem

Claude loads a folder, and with it every skill in that namespace
Hunting for a skill by reading folders loads several folders: redundant context

The fix: an index over MCP

ai-rulez indexes the repo's skills; find_skill returns ranked, scored matches
load_skill returns the one skill needed, nothing else
A file watcher can keep the index current (off in our config)

Task

“details for this customer?”

find_skill

ranked, scored matches under the product's path

Choose

the best match for the task

load_skill

only that skill enters the context

• Layering keeps things contained. Discovery is the other half: finding the right skill without loading everything. And it has a problem.

• Once Claude loads a folder, it loads every skill in that namespace. A folder with a dozen skills puts a dozen descriptions into the context, whether the task needs one of them or none. And if the agent hunts for the right skill by reading folders, it loads several folders, and a lot of redundant context with them.

• The fix is an index. ai-rulez, which is a tool I authored and is open source, builds an index of the repository's skills and serves it over MCP, the protocol Claude Code uses to talk to tool servers. It exposes two tools: find skill, and load skill.

• Find skill takes the task and, optionally, a path. It returns ranked results, each with a score, so the agent sees the best candidates and how well they match. Load skill then returns one skill's body. The session loads exactly the skill it needs, and nothing else from that namespace.

• The index is built when the server starts, and ai-rulez can keep it current with a file watcher. In our configuration the watcher is switched off, so a new skill shows up after the server restarts.

• The root CLAUDE.md tells the agent to scope the search to the product's path. How the agent learns the product is the next topic, in the results.

• As you'll see in the results, scoping the search is what moved recall.

10 of 14

EVALS I

Two layers of tests: static and dynamic

Static analysis using ai-rulez

free · offline · in CI

Calculates the loading and layering, checking what prompt is composed at each stage.
What tokens a session sees at start? What terms enter context?
Asks: what will it see?

Dynamic evaluation: AI as a judge

local · tens of dollars a run

230 YAML cases, each with criteria
Real hermetic claude -p sessions that must answer using skills, measuring recall ("did it use what it needed?") and precision ("was everything it used appropriate?")
Asks: what does it do with it?

Tune on one split of the cases; a held-out split gives the verdict

• How do we know it works? We test this out using two layers.

• The first layer is static analysis, using ai-rulez. It checks the files and what a session will see when it starts. It is offline and deterministic, so it runs in CI on every config change. It answers: what is in the context before the agent has read a single file, how many tokens is that, and does any forbidden product term appear in it?

• The second layer is dynamic evaluation using AI as a judge. We created a testing harness, which uses a Claude Opus orchestrator, and launches a fleet of Sonnet model’s in headless session, each sandboxed in a discardable git checkout without access to the actual test cases. The session has to answer using the skillsit has access to.

• Then we score the transcript. Some checks are deterministic: did it load the right skill, did it mention the wrong product. For the answer itself, a judge model grades it against the case's criteria.

• What does that measure? Whether sessions find the right skill, stay inside their product, and give a good answer. How precise is it? We checked the judge against a second model, and I'll show those numbers in a moment. What does it cost? Real money: tens of dollars for a judged run, and minutes. So it runs locally, on demand, not in CI.

• The two layers answer different questions. Static asks: what will the session see? Dynamic asks: what does it do with it? And we tune on one split of the cases, while a held-out split gives the verdict.

11 of 14

EVALS II

The static layer is ai-rulez

ai-rulez 5.10.0, pinned in CI and in the harness; a test fails if the pins diverge.

ai-rulez exposure

what a session sees at start, in tokens per phase

exposure leakage

product terms at start; a CI gate that exits 2

exposure collisions

skills in different areas with overlapping listings

ai-rulez validate

the same rules as findings, in CI

areas.toml

the area map: 19 areas with paths, terms and ambiguous words

validate findings

A product term at session start (error)

An ambiguous word with no product term beside it

An acceptance that accepts nothing

7 → 0

product terms at start, kept at 0 by CI

8 accepted, each with a written reason

• Static first, and this is where ai-rulez is the engine. It has a command family called exposure, and it models what a Claude Code session loads.

• Plain exposure shows what a session sees at start, in tokens, split by phase: at start, on demand, and manual. Exposure leakage lists product terms in that start context. It's a CI gate: it exits with code 2 if it finds one. Exposure collisions finds skills in different areas whose listings overlap, which is how one product's skill ends up answering another's question.

• ai-rulez validate, the lint step that we run in CI, also checks the same session-exposure rules that ai-rulez exposure reports. In validate they show up as ordinary lint findings, with a file, a line and a severity. So no separate tool or job is needed.

• The result: product terms at session start went from seven to zero. And CI keeps it at zero. Eight findings are accepted on purpose, mostly names in the products map, each with a written reason.

12 of 14

EVALS III

Why trust the numbers

Example case

prompt: show me this customer's details

asker: Product B developer

must_use: b-customer-lookup

must_not: any Product A skill

behaviour: answer, or ask which product

criteria:

- customer means a Product B account

- never uses Product A's records

Regex checks

skill loaded · no forbidden term · command ran

Judge grades

is the answer right, per this case's criteria?

κ 0.669

agreement beyond chance, answers

κ 0.847

agreement beyond chance, criteria

86%

broken answers caught

93%

good answers passed

Decision rule and a 6-read holdout budget, fixed in advance · Sonnet sessions, Opus judge

• Dynamic evals, and why you should believe the numbers.

• Here's one case. A prompt, who's asking, the skill that must be used, what must stay out, whether the right move is to answer or to ask, and criteria for a good answer.

• Some of that a regex can check: the right skill was loaded, a forbidden product term is absent, the right command ran. But whether the answer is actually right for this case, a regex can't tell. So a judge model grades each answer against the case's criteria, with reference facts in front of it. Sonnet runs the sessions and Opus judges them. That lets us grade hundreds of answers consistently and repeatedly, and verdicts are cached, so re-scoring is free.

• But a judge is a model, so we measure how far to trust it. A second model re-grades a sample: agreement beyond chance, a kappa, is 0.669 on answers and 0.847 on criteria. And mutation tests: we break good answers on purpose, and the judge fails 86 percent of the broken copies and passes 93 percent of the good ones. Be careful: these are indications, not measurements against human labels, and two models can share a bias.

• The other protection is procedural. The decision rule is fixed in advance: a significant gain, corrected for multiple tests, no regression on guardrails, and a start-token budget. And the held-out cases can be read only a limited number of times, six, each logged in a ledger. So tuning can't quietly overfit the verdict.

13 of 14

RESULTS

What we tried: a staircase of results

Held-out: 53 cases, two runs each; user-specific context covers 39 of them. Wrong-product answers rest on 5 cases.

Wording tweaks

read the guide first, hoist shared skills

no gain

Start-hook skill list

a cross-product menu, +16k start tokens

dropped

Scoped discovery

+31 recall (tune split); small on held-out

kept

User-specific context

developer’s product, confirmed once, in a local file

opt-in

master

layered design

+ user context

Pass rate

39%

81%

96%

Right skill found

22%

72%

94%

Wrong-product answers

70%

/init-command saves a developer’s product, in a git-ignored CLAUDE.local.md loaded at every session start

• Now, what did we actually try? We let the evals decide.

• First, wording: tell the agent to read the product guide first, hoist shared skills back to the root. Sensible edits, no measurable gain. The problem was mechanical: product skills load only when a session reads a file in that folder, and searching doesn't count.

• Second, a session-start hook listing every product's skills. Recall went up, but the agent chose from a menu of all products before it knew its own, and every start cost about sixteen thousand more tokens. We dropped it, on judgement, not on a significant result.

• Third, scoped discovery through ai-rulez. On the tuning cases that was thirty-one recall points. But careful: those cases leaned toward regressed areas, with one run each, and on held-out cases discovery alone added only a few points.

• Fourth, user-specific context. An init command, armis-init, looks up your GitHub teams and the products they own in CODEOWNERS, you confirm one, and it records it in your own git-ignored local file. Claude Code loads that file at every session start, so the session knows your product. Without it, the root file makes the agent ask which product, because there is no default.

• So here is the fair picture, on held-out cases, step by step. Pass rate 39, 81, 96. Right skill found 22, 72, 94. Wrong-product answers 70, 0, 0.

• Answer quality is not on the slide, on purpose. The design alone did not improve it, it even dipped a little, and I can't tell you why. With user-specific context it rose about fifteen points. The design bought containment and discovery. Quality needed the product.

• Reading transcripts, not just scores, also caught a bug in our own harness: a printed spend line read like money running out, so the model quit early. The biggest loss left is over-asking, the agent asking which product when the prompt named it. That's the next target.

1 of 14

2 of 14

3 of 14

4 of 14

5 of 14

6 of 14

7 of 14

8 of 14

9 of 14

10 of 14

11 of 14

12 of 14

13 of 14

14 of 14