A I   T A L K S   R O A D M A P

MASTER AI ENGINEERING WITH THESE VIDEOS

Focus: learn from seven of the best AI engineering talks, understand every idea in them, and turn each into skill you can show.

Inside

GUIDE    How to use this roadmap and the learning order

TALK 1   Error Analysis (Andrew Ng)

TALK 2   Building and Evaluating Agents (Sayash Kapoor)

TALK 3   How We Build Effective Agents (Barry Zhang)

TALK 4   Do Not Build Agents, Build Skills (Zhang and Murag)

TALK 5   12-Factor Agents (Dex Horthy)

TALK 6   No Vibes Allowed (Dex Horthy)

TALK 7   Beating RL With Reflection: GEPA (Lakshya Agrawal)

GUIDE    How to be better, capstone, interview bank, tools

 Every talk: the big idea, every concept explained, tools with links, a hands-on exercise, mistakes to avoid, resources and interview questions.

How to use this roadmap and the learning order

This roadmap turns seven well-known AI engineering talks into a structured course. For every talk you get the big idea in plain words, every important concept explained, the tools mentioned or closely related with real links, a hands-on exercise, common mistakes, ways to get better, interview questions with what a strong answer covers, and further resources.

An honest note on sources. This document is built from each talk's title, description and the public material around it (the speakers' own articles, repositories, papers and courses). It is not a transcript. Where a number or a slide detail matters, check it against the video itself. Every link below points to a real, public page.

The seven talks, in the order you shared them:

Your original links, for reference. Match them to the titles above by opening each one:

• https://www.youtube.com/watch?v=8kMaTybvDUw
• https://www.youtube.com/watch?v=rmvDxxNubIg
• https://www.youtube.com/watch?v=D7_ipDqhtwk
• https://www.youtube.com/watch?v=CEvIs9y1uog
• https://www.youtube.com/watch?v=d5EltXhbcfA
• https://www.youtube.com/watch?v=JoAxZsdw_3w
• https://www.youtube.com/watch?v=OA-Mc60Rboo

The recommended learning order

Do not watch these in the order of popularity. Watch them in the order that builds understanding. The talks are not independent: each answers a question the previous one raises.

The one idea behind all seven talks

If you remember only one sentence from this whole roadmap, make it this: an AI system is only as good as the context it sees, the loop it runs in, and the measurements you use to improve it. Context engineering decides what the model sees. Agent design decides the loop. Evals and error analysis decide how you improve. Skills and 12-factor patterns make the result maintainable. GEPA automates part of the improvement.

A 7 week study plan

T A L K   1   ────      1 / 7

Carrying Out Error Analysis

Focus: Learn to find out which mistakes are worth fixing before you spend a week fixing the wrong one.

Why this talk matters

Every AI team has the same temptation: the system gets something wrong, someone has a theory, and a week later a fix ships that barely moves the metric. Error analysis is the discipline that stops this. It is a manual, unglamorous process of looking at real failures and counting them by cause so that you can see where the biggest win is.

It is arguably the most transferable skill in this entire roadmap. It was taught for classical machine learning, yet it is exactly what modern LLM practitioners now call "looking at your data" and "reading your traces". Hamel Husain, Shreya Shankar and others teaching LLM evals describe the same loop: sample outputs, label what went wrong, group failures, fix the biggest group.

The big ideas explained

Idea 1: Do not fix what you have not measured. Suppose your cat classifier has 10 percent error and you think dogs mislabelled as cats are a problem. Before building a dog-specific fix, take 100 mistakes and count how many are actually dogs. If only 5 of the 100 are dogs, the most you could ever gain from a perfect dog fix is removing 5 percent of your errors, taking 10 percent error down to 9.5 percent. That number is called the ceiling for that fix. If 50 of the 100 are dogs, the ceiling is far higher and the fix is worth a week.

Idea 2: Use a spreadsheet, not a model. Take a sample of mistakes (Ng suggests starting with around 100 misclassified examples). Make one row per example and one column per candidate cause, such as "blurry image", "dog mistaken for cat", "great cat variety", "mislabelled in the dataset". Tick the boxes. Add a free-text comment column so new categories can appear as you read. At the bottom, total each column. The percentages tell you where to spend time.

Idea 3: Evaluate several ideas in parallel. One pass through the same spreadsheet lets you compare many ideas at once. This is faster than trying each idea as a separate experiment.

Idea 4: Categories are discovered, not decided up front. You begin with a few guesses, and as you read examples new categories appear. Add columns as you go. This is the same idea as "open coding" in qualitative research, and it is why people who skip reading the data miss the real failure modes.

Idea 5: Incorrectly labelled data is an error category too. If your ground truth is wrong, your metric is lying. Count how many mistakes are really labelling mistakes. Deep learning models are fairly robust to random label noise in the training set, but label errors in your dev or test set directly corrupt your measurement, so fix those first when they form a significant share of errors.

Idea 6: Look at examples the model got right as well. Fixing errors can break things that already work. Sampling some correct cases shows whether a fix could introduce new problems.

Idea 7: Look at the data by hand, always. Ng stresses that a few hours of looking at examples is among the highest-return activities in a project. Automation comes after understanding.

Concepts glossary

How this applies to LLM and agent systems today

Replace "image" with "trace". An LLM application's equivalent loop looks like this:

1. Collect 50 to 100 real or realistic traces, especially the ones users thumbs-downed or that failed a check.
2. Write one short note per trace about what went wrong, in your own words, without forcing categories.
3. Group the notes into 5 to 10 named failure modes (for example "retrieved the wrong document", "ignored a formatting instruction", "tool called with a wrong argument").
4. Count each mode. Fix the biggest, then re-measure on the same set.
5. Turn the most important failure modes into automated evals so they are caught forever after.

Tools and links

Hands-On Exercise

Do this step by step

1. Pick one system and one clear quality question, such as "is the answer correct and grounded".
2. Gather 50 failing outputs. If you have no real users, generate 100 varied inputs and keep the ones that fail.
3. Read each failure slowly and write a one line note. Do not use categories yet.
4. After about 30 notes, cluster them into named failure modes. Keep an "other" column.
5. Count. Rank by frequency and by how easy the fix is.
6. Fix the top mode. Re-run the same inputs. Record the new count.
7. Convert that failure mode into an automated check so it cannot silently return.

Acceptance criteria: a spreadsheet with at least 50 rows, at least 5 named failure modes with counts, one fix shipped, and a measured before and after.

Common mistakes

• Starting with a theory and only looking for examples that confirm it.
• Looking at 5 examples and calling it analysis. Aim for at least 50 to 100.
• Using generic categories like "wrong answer". Be specific enough that a category suggests a fix.
• Never re-measuring after a fix.
• Letting an LLM do all the labelling without ever reading traces yourself. LLM help is fine after you have read enough to define the categories.

How to get better at this

• Set a recurring 30 minute "trace reading" slot every week. Habits beat heroic efforts.
• Keep a living failure-mode document and update counts after each release.
• Ask a teammate to label the same 20 examples and compare. Disagreements reveal vague categories.
• Learn the vocabulary of bias, variance and data mismatch from the full course.
• Read Hamel Husain's writing on evals and Eugene Yan's patterns post, listed below.

Interview questions

1. What is error analysis and why do it before changing a model or prompt? Strong answer: manual review of failures grouped by cause so that effort goes to the largest and most fixable category, avoiding work whose ceiling is tiny.
2. Your system has 10 percent error and 5 percent of those errors are dogs. Should you build a dog fix? Strong answer: the ceiling is a 0.5 point drop (10 percent to 9.5 percent), so only if it is very cheap. Look for bigger categories first.
3. How many examples should you review and how do you choose them? Strong answer: start with around 50 to 100 random failures, stratify by user segment or input type if you have them, and keep going until no new categories appear.
4. How do you handle mislabelled data discovered during analysis? Strong answer: count it as its own category, fix labels in dev and test sets first because they corrupt measurement, and be careful to apply the same correction rule to all examples.
5. How does error analysis translate to LLM applications? Strong answer: read traces, write open notes, group into failure modes, count, fix the biggest, then encode it as an automated eval.
6. What is the difference between error analysis and an eval suite? Strong answer: error analysis is exploratory and discovers what to measure; evals are automated checks that keep measuring it.
7. How do you decide between fixing the largest category and the easiest one? Strong answer: estimate impact times ease, and consider whether the fix is a prompt tweak, a retrieval change or a model change.
8. Why can optimising a metric without error analysis go wrong? Strong answer: the metric may not capture the real user pain, and improvements may come from a category that was never the bottleneck.

Resources

• Machine Learning Yearning by Andrew Ng (free book with the same method)
• Structuring Machine Learning Projects on Coursera
• Deep Learning Specialization
• Your AI Product Needs Evals by Hamel Husain
• Patterns for Building LLM-based Systems by Eugene Yan
• Langfuse documentation on annotation and datasets

T A L K   2   ────      2 / 7

Building and Evaluating AI Agents

Focus: Stay realistic about what agents can do, and learn how to measure whether yours actually works.

Why this talk matters

Agents are the most hyped idea in AI, and the gap between demos and daily reliability is large. Sayash Kapoor is a Princeton researcher and co-author, with Arvind Narayanan, of the book AI Snake Oil and the newsletter of the same name. His talk, as its slide title "Stop Over-hyping Agents" suggests, argues for sober thinking about what agents can do and for better evaluation.

For an engineer, this talk is a protection against two expensive mistakes: building something the technology cannot yet do reliably, and shipping something you cannot measure.

The big ideas explained

Idea 1: A demo is not a product. A system that works in a controlled demo may fail on a large share of real tasks. Products such as the Rabbit R1 and the Humane AI Pin, which the talk uses as examples of over-hyped agent-style devices, showed how large the gap can be between launch claims and real use.

Idea 2: Capability is not reliability. A model that can solve a task once may fail it one time in five. For real work, users need the thing to succeed nearly every time. Each extra step in an agent loop multiplies the chance of an error, so long tasks compound failure. For example, 10 steps at 95 percent success each succeed together only about 60 percent of the time.

Idea 3: Evaluating agents is harder than evaluating models. A model benchmark gives one input and checks one output. An agent takes many steps, uses tools, changes state in an environment and may take different routes to the same answer. You need to judge outcomes, trajectories and cost together.

Idea 4: Always report cost with accuracy. Kapoor's research group (AI Snake Oil and the Princeton HAL project) showed that agent accuracy can often be bought by running more attempts. A fair comparison plots accuracy against dollar cost, because a slightly more accurate agent that costs 20 times more is often not better. Their paper "AI Agents That Matter" makes this argument with data.

Idea 5: Reasoning models change the picture but do not remove the problem. Models trained to think step by step make agents more capable, yet they do not fix hallucination, tool misuse or evaluation gaps.

Idea 6: Benchmarks leak and saturate. If a test set ends up in training data, scores climb without real capability gains. Holdout sets, fresh tasks and checking for contamination matter.

Idea 7: Build an evaluation pipeline for your own task. Public benchmarks tell you little about your use case. Collect your own examples, define success criteria, and use human review plus LLM judges that you have validated. Tools in the EvalGen line of research help you align an LLM judge with your own preferences by letting you grade a few outputs and then checking the judge against them.

Idea 8: Think in terms of consequences. Mistakes in a low-stakes assistant are fine. Mistakes in agents that spend money, send emails or change production data are not. Decide the cost of an error before choosing how autonomous the agent can be.

Concepts glossary

Tools and links

Hands-On Exercise

Do this step by step

1. Write 30 realistic tasks with a clear definition of success for each.
2. Prefer checks you can run in code (a file exists, a SQL result matches, an API state changed). Use an LLM judge only for fuzzy qualities.
3. Grade 20 outputs yourself, then measure how often your judge agrees with you. Adjust the judge prompt until agreement is high.
4. Run each task 5 times. Compute average success and the share of tasks that pass all 5.
5. Record tokens and dollar cost per task.
6. Plot accuracy against cost for two variants (for example a cheaper and a stronger model).
7. Write three lines on what you would not trust this agent to do.

Acceptance criteria: reliability and cost reported next to accuracy, a judge with measured agreement against your own labels, and a clear "do not use for" list.

Common mistakes

• Reporting one lucky run as accuracy.
• Comparing agents without controlling cost or number of attempts.
• Trusting an LLM judge that was never checked against human labels.
• Testing only on public benchmarks.
• Letting agents take irreversible actions without approval.

How to get better at this

• Read the AI Snake Oil newsletter and the "AI Agents That Matter" paper.
• For every agent you build, write the "stop conditions" and the "never do" list first.
• Track reliability (pass^k), not just average accuracy.
• Learn basic statistics: confidence intervals, sample size, variance across runs.
• Practise explaining limits to non-technical stakeholders. Honest expectation setting is a senior skill.

Interview questions

1. Why do agents that look great in demos disappoint in production? Strong answer: demos are curated, production inputs are messy, per-step errors compound, and evaluation was too narrow or contaminated.
2. Explain compounding error with an example. Strong answer: ten steps at 95 percent each give roughly 60 percent end to end, so shorten chains, add checks and verification steps.
3. What is the difference between pass@k and pass^k and which matters for products? Strong answer: pass@k is any success in k tries, pass^k is all k succeed. Products need pass^k style reliability.
4. Why report cost alongside accuracy? Strong answer: accuracy can be bought with more compute, so the fair comparison is the accuracy and cost frontier.
5. How do you validate an LLM judge? Strong answer: have humans label a sample, measure agreement, inspect disagreements, refine the rubric, and recheck periodically.
6. How would you evaluate a customer support agent? Strong answer: build a task set from real tickets, check outcomes (correct resolution, policy compliance), measure cost, test repeated runs and adversarial users.
7. What is benchmark contamination and how do you avoid it? Strong answer: test data appearing in training data. Use private holdouts, fresh tasks and dated problems.
8. When should a human stay in the loop? Strong answer: when error cost is high or actions are irreversible, and during early rollout while reliability is being measured.

Resources

• AI Snake Oil newsletter
• AI Snake Oil, the book, by Arvind Narayanan and Sayash Kapoor
• AI Agents That Matter (paper)
• Who Validates the Validators? (EvalGen paper)
• Inspect documentation
• Hamel Husain on evals

T A L K   3   ────      3 / 7

How We Build Effective Agents

Focus: Learn when an agent is the right tool and how to design the simplest one that works.

Why this talk matters

Barry Zhang works on agents at Anthropic and co-wrote, with Erik Schluntz, the widely read article "Building Effective Agents". The talk compresses its message into three ideas: do not build an agent for everything, keep the design simple for as long as you can, and think like your agent. It is the clearest single guide to agent design from the people who build the underlying models.

The big ideas explained

Idea 1: The evolution from simple to complex. Teams moved from single LLM calls, to workflows (several calls chained in code), to agents (the model decides the steps). More autonomy gives more capability, and also more cost, latency and risk. Start at the left and move right only if you must.

Idea 2: Workflows versus agents. Anthropic's article defines a workflow as an LLM and tools orchestrated through predefined code paths, and an agent as a system where the LLM directs its own process and tool use. Common workflow patterns are prompt chaining, routing, parallelisation, orchestrator-workers and evaluator-optimiser.

Idea 3: The checklist for "should I build an agent". As presented in the talk, four questions:

• Task complexity. Is the path to the answer unclear or open ended? If you can draw the decision tree, build a workflow instead.
• Value of the task. Is it valuable enough to justify the tokens an agent will burn? A task worth cents does not justify an expensive exploration.
• Doable parts. Are the critical capabilities (reading the codebase, using the tools, reasoning about the problem) actually achievable by the model? If not, narrow the scope first.
• Cost of error and discovery of errors. If mistakes are costly and hard to detect, keep a human in the loop or limit autonomy.

Coding is the example where agents shine: the problem is open ended, the value is high, models are good at it, and the output can be verified with tests.

Idea 4: Keep it simple. An agent is a model using tools in a loop. Three components define it: the environment the agent operates in, the tools that let it act and receive feedback, and the system prompt that sets goals and rules. Build and iterate on these three before adding clever machinery. Complexity can be added later, after you have a baseline that works.

Idea 5: Think like your agent. An agent only sees what is in its context window: your prompt, tool descriptions and results. It does not know what you know. Zhang suggests imagining yourself in the model's seat with only a screenshot and a vague tool, and noticing how little information you have. Then ask the model itself: is this instruction ambiguous, are these tool descriptions clear, what would help you decide.

Idea 6: Open questions the talk raises. Making agents budget aware (limits on tokens, time and money), self-evolving tools, and multi-agent collaboration. These are active research areas, so treat them as experiments, not defaults.

Idea 7: Tools deserve as much design effort as prompts. Anthropic's engineering team later published guidance on writing tools for agents. Clear names, narrow responsibilities, helpful error messages and token-efficient outputs make agents far more reliable.

Concepts glossary

Tools and links

Hands-On Exercise

Do this step by step

1. Pick a task and answer the four checklist questions in writing. If the answer is "no" to two or more, build a workflow.
2. Write the three components: environment, tools and system prompt.
3. Implement the loop yourself with the raw API: send messages, read tool calls, run them, return results, stop on a final answer.
4. Add a maximum step count and a token budget.
5. Test "think like your agent": print exactly what the model sees at each step and read it as if you were the model.
6. Rewrite one tool description to remove ambiguity and re-measure.
7. Only then add a framework or more complexity, and only if you can show it helps.

Acceptance criteria: a working agent loop, a documented checklist decision, a step and cost limit, and a measured comparison against the workflow version.

Common mistakes

• Building an agent where a three step workflow would do.
• Reaching for a framework before understanding the loop it hides.
• Vague tool names and descriptions that the model misreads.
• No stopping condition, causing endless loops and cost spikes.
• Forgetting that the agent cannot see anything you did not put in its context.

How to get better at this

• Read "Building Effective Agents" twice, then re-implement each workflow pattern by hand.
• Read transcripts of your agent runs every week, the way Ng recommends reading errors.
• Study open source agents such as the Claude Agent SDK examples and read how they structure tools.
• Practise cutting features. Better agents are often simpler agents.
• Learn about context management: truncation, summarisation and memory.

Interview questions

1. When should you build an agent instead of a workflow? Strong answer: when the path is open ended, value justifies cost, capabilities are within reach, and errors are detectable or low cost. Otherwise use a workflow.
2. Define an agent in one sentence. Strong answer: a model using tools in a loop, deciding its own next step, until a goal or stop condition is met.
3. Name three workflow patterns and when to use each. Strong answer: chaining for sequential subtasks, routing for distinct input categories, parallelisation for independent subtasks or voting. Mention orchestrator-workers and evaluator-optimiser too.
4. What does "think like your agent" mean in practice? Strong answer: inspect exactly what is in the context, read tool descriptions from the model's seat, and ask the model what is unclear.
5. How do you stop an agent from running forever? Strong answer: step limit, token and time budget, explicit stop tool, and detecting repeated actions.
6. Why are tool descriptions so important? Strong answer: the model chooses tools based on name and description alone, so ambiguity causes wrong calls. Describe purpose, parameters, examples and failure messages.
7. How do you handle high cost-of-error tasks? Strong answer: limit autonomy, add approvals, use read-only tools first and verify results before irreversible actions.
8. What are the trade-offs of multi-agent systems? Strong answer: parallelism and separation of concerns versus coordination overhead, higher token cost and harder debugging. Use only when a single agent clearly hits limits.

Resources

• Building Effective Agents (Anthropic engineering)
• Writing effective tools for agents
• Effective context engineering for AI agents
• How we built our multi-agent research system
• Anthropic Cookbook agent patterns
• LLM Powered Autonomous Agents by Lilian Weng

T A L K   4   ────      4 / 7

Don't Build Agents, Build Skills Instead

Focus: Give one general agent expert knowledge by writing skills (folders of instructions and scripts) instead of building a new agent for every domain.

Why this talk matters

The talk argues that the thing worth building is no longer a custom agent per domain. Agents have converged on a similar core: a model with a bash tool and a filesystem, running a loop. What differs between a finance agent and a legal agent is the knowledge they carry. Agent Skills package that knowledge as folders.

Seen this way, the work of an AI engineer shifts from building scaffolding to capturing procedural expertise, which is exactly what experts and domain teams are good at.

The big ideas explained

Idea 1: Scaffolding is converging. Whether you look at coding agents or general agents, they tend to have the same parts: a loop, a filesystem, a shell, a few tools, and a way to load context. A general agent with a computer can do a surprising range of work, so rebuilding that harness per use case is wasteful.

Idea 2: Intelligence is not the same as expertise. A brilliant generalist given a complicated tax form will still do it poorly without domain procedures. Skills carry the procedures, templates and gotchas that make output reliable.

Idea 3: What a skill is. A skill is a directory containing a `SKILL.md` file with a name, a description and instructions, plus optional scripts, reference files and assets. The agent sees only the short name and description at startup. When a task matches, it reads the full instructions, and it opens extra files only when needed.

Idea 4: Progressive disclosure. This is the key design idea. Instead of stuffing every instruction into the context window, the agent loads information in layers: metadata first, then the main instructions, then supporting files, then script outputs. Context stays small and relevant, so you can install many skills without penalty.

Idea 5: Skills can contain code. A skill can bundle scripts the agent runs (for example a PDF form filler or a data validator). Running a script is deterministic and cheaper than having the model write the same code each time. This is the answer to "the model keeps rewriting the same helper".

Idea 6: Skills and MCP are complementary. MCP (Model Context Protocol) gives the agent connections to external systems such as databases, issue trackers and APIs. Skills teach the agent what to do with those connections. MCP is the plumbing and the tools; skills are the know-how.

Idea 7: Anyone can write one. Because a skill is a folder of text and scripts, domain experts can contribute without writing agent code. Skills can be versioned in git, shared with a team and improved over time. The talk imagines a growing ecosystem of shared skills and agents that learn new ones continuously.

Idea 8: Security needs care. Skills can run code, so treat them like software you install. Read them, use trusted sources and sandbox execution.

Concepts glossary

What a good SKILL.md looks like

A minimal skill folder:

• `my-skill/SKILL.md` with a name and a description that says exactly when to use it
• `my-skill/reference.md` for long background that is only read when needed
• `my-skill/scripts/validate.py` for deterministic steps

The description is the most important line. It is what the agent matches against the user's request, so write it as "Use when the user asks for X, Y or Z" and mention the file types or keywords involved.

Tools and links

Hands-On Exercise

Do this step by step

1. List five tasks you repeat that a general assistant does inconsistently.
2. Pick one. Write down the exact steps you follow, the template, and the three mistakes people usually make.
3. Create the folder and SKILL.md. Write a sharp description with trigger words.
4. Move long reference material to a separate file and link to it from SKILL.md.
5. Move any mechanical step (parsing, validation, formatting) into a script and tell the skill when to run it.
6. Test with five phrasings of the request, including vague ones. Check the agent loads the skill and follows it.
7. Improve based on failures, then commit the folder to git.

Acceptance criteria: description triggers on at least 4 of 5 test prompts, no instruction exceeds what is needed, scripts are deterministic and tested.

Common mistakes

• A vague description, so the skill never triggers.
• One giant SKILL.md that wastes context instead of linking to reference files.
• Putting logic in prose that should be a script.
• Installing unreviewed third-party skills that run code.
• Duplicating tool descriptions that MCP already provides.

How to get better at this

• Read every skill in the Anthropic skills repository and notice how descriptions are written.
• Treat skills like code: tests, version control, reviews.
• Learn when to choose a skill, an MCP server or a subagent. A rule of thumb: skill for procedures, MCP for access, subagent for isolated context.
• Capture expertise from colleagues by interviewing them while they do the task, then write it down.

Interview questions

1. What is an Agent Skill and how is it different from a prompt? Strong answer: a reusable folder with instructions, scripts and references that the agent loads on demand, unlike a prompt that always occupies context.
2. Explain progressive disclosure. Strong answer: metadata first, full instructions on match, extra files only when needed. It keeps context small and scales to many skills.
3. How do skills and MCP differ and work together? Strong answer: MCP connects to systems and exposes tools; skills teach procedure and judgement. Use skills to instruct how to use MCP tools well.
4. Why include scripts in a skill? Strong answer: determinism, speed, lower token cost and no re-generation of the same code.
5. What makes a good skill description? Strong answer: states when to use it, names inputs and outputs, includes trigger words, stays short.
6. What security risks do skills introduce? Strong answer: arbitrary code and instruction injection. Review, restrict sources, sandbox and limit permissions.
7. Why argue "build skills, not agents"? Strong answer: the harness is converging, so differentiation moves to domain knowledge, which is cheaper to author, share and maintain as skills.
8. How would you evaluate a skill? Strong answer: a prompt set with expected behaviour, trigger accuracy, output quality checks and regression runs after edits.

Resources

• Agent Skills overview (docs)
• Equipping agents for the real world with Agent Skills
• Anthropic skills repository
• Model Context Protocol introduction
• Building Effective Agents

T A L K   5   ────      5 / 7

12-Factor Agents

Focus: Turn agent demos into reliable software by applying twelve engineering principles.

Why this talk matters

Dex Horthy talked to many founders building agents and found that the products that actually reached customers rarely used a big agent framework end to end. They were mostly ordinary software with small, well-controlled LLM steps. He captured the patterns in the open source repository 12-factor-agents, named after the classic "Twelve-Factor App" methodology for web services.

The talk is a design guide for reliability. It also introduced the phrase "context engineering" to a wide audience, meaning that you carefully shape what goes into the model's context window.

The big ideas explained

The repository lists twelve factors. The summaries below are in plain words. Check the repository for the exact wording.

Factor 1: Natural language to tool calls. The most reliable thing LLMs do in an agent is turn a human request into a structured call, such as a JSON object naming a function and its arguments. Your code then executes it.

Factor 2: Own your prompts. Do not hide prompts inside a framework. Write them as code you can read, test, version and tune. Prompts are your product's logic.

Factor 3: Own your context window. You decide exactly what the model sees. You do not have to use a standard message format. You can pack events, errors and state in your own compact format to improve results and save tokens. This is context engineering.

Factor 4: Tools are just structured outputs. A "tool call" is simply the model emitting structured data. What you do with it is ordinary code, which means you can validate it, branch on it and test it.

Factor 5: Unify execution state and business state. Avoid keeping a separate hidden agent state. Make the history of events the single source of truth so that you can pause, resume and inspect everything.

Factor 6: Launch, pause and resume with simple APIs. Agents should be startable, pausable and resumable from outside, for example through webhooks, so they can wait for slow tools or human answers without holding a process open.

Factor 7: Contact humans with tool calls. Make "ask a human" one more structured tool call, such as requesting clarification or approval. This turns human-in-the-loop from an afterthought into a normal step. HumanLayer's related ideas about agent-to-human communication are in its open source SDK.

Factor 8: Own your control flow. Decide in code what happens after each model output: loop, break, retry, wait for approval or call a different step. Do not surrender the loop to a framework you cannot change.

Factor 9: Compact errors into the context window. When a tool fails, summarise the error and feed it back so the model can recover. Cap retries and clear old errors so they do not clutter the context.

Factor 10: Small, focused agents. Agents that handle 3 to 20 steps with a narrow scope stay on track. Large open ended agents lose focus as context grows. Embed small agents inside a deterministic workflow.

Factor 11: Trigger from anywhere and meet users where they are. Let agents start from email, Slack, cron jobs or events, not only a chat box.

Factor 12: Make your agent a stateless reducer. Think of the agent as a function that takes the current event history and returns the next step, without hidden memory. Stateless functions are easy to test, replay and scale.

Why this approach works

• Debuggability. With a visible event log, you can replay any run.
• Testability. Pure functions and structured outputs can be unit tested.
• Cost and speed. You pack only what matters into the context.
• Safety. Humans approve high risk actions through the same interface as every other tool.
• Flexibility. You can swap models, frameworks or storage without rewriting the product.

Concepts glossary

Tools and links

Hands-On Exercise

Do this step by step

1. Define a typed schema for each possible next step (for example reply, archive, ask human, done).
2. Write your prompt in a plain file. Version it with git.
3. Write a function that turns the event list into your own compact context string.
4. Call the model, parse the result into a typed step, and branch on it in your own loop.
5. Add a step type "ask human". When chosen, save state and stop. Provide an endpoint to resume with the human's answer.
6. Catch tool errors, compact them into one short event, and retry at most three times.
7. Replay a saved event log to reproduce a bug without calling the model.

Acceptance criteria: no framework controls the loop, every run is replayable from its event log, humans approve irreversible actions, and the agent has a retry cap.

Common mistakes

• Using a framework as a black box and then being unable to debug it.
• Dumping the entire raw history into the context.
• Retrying failed calls forever.
• Giving one agent a huge scope instead of several small ones.
• Putting business state in one place and agent state in another so they drift apart.

How to get better at this

• Read the repository end to end and rebuild one factor per day for 12 days.
• Practise writing prompts as code with tests.
• Learn durable execution concepts (Temporal, queues, idempotency).
• Study the original Twelve-Factor App methodology at 12factor.net to see the parallels.

Interview questions

1. What does "own your context window" mean? Strong answer: you control the exact content and format sent to the model, optimising for information density, clarity and cost instead of relying on default message arrays.
2. Why treat tool calls as structured output? Strong answer: it separates model decisions from code execution, allowing validation, branching, testing and safer handling.
3. What is a stateless reducer agent and why is it useful? Strong answer: the agent maps event history to the next step with no hidden state, making runs replayable, testable and easy to scale.
4. How should human approval be modelled? Strong answer: as a normal tool call that pauses the run, stores state and resumes through an API or webhook.
5. Why prefer small focused agents? Strong answer: context grows and quality falls with long loops, so narrow agents inside deterministic workflows are more reliable.
6. How do you handle errors in an agent loop? Strong answer: compact the error, add it to context so the model can adapt, cap retries and escalate to a human.
7. What are the risks of heavy agent frameworks? Strong answer: hidden prompts and control flow, hard debugging and lock-in. Use them only if they leave you in control.
8. How would you make an agent idempotent? Strong answer: use idempotency keys for side effects, record completed steps in the event log and design tools so a repeat call is harmless.

Resources

• 12-factor-agents repository
• The Twelve-Factor App
• HumanLayer on GitHub
• Effective context engineering for AI agents (Anthropic)
• BAML documentation

T A L K   6   ────      6 / 7

No Vibes Allowed: Complex Codebases

Focus: Make coding agents solve hard problems in large, messy codebases through research, planning and context discipline.

Why this talk matters

"Vibe coding" works for prototypes. In large existing codebases it often produces slop, rework and code that other engineers reject. The talk, as its description and slides indicate, draws on observations from many engineering teams and a widely discussed Stanford study of AI coding productivity. The headline finding people took from it: AI tools raise output on greenfield, simple tasks, while on complex work in mature codebases the gains shrink and rework can eat them.

Horthy's answer is not "wait for better models". It is a workflow that gets more from today's models by managing context carefully.

The big ideas explained

Idea 1: Why complex codebases break agents. The model's quality drops as the context fills up with irrelevant text, failed attempts and noise. Horthy sometimes calls the zone where quality degrades the "dumb zone". The aim is to keep the working context small, relevant and accurate.

Idea 2: Context engineering and harness engineering. Context engineering is deciding what goes into the context. Harness engineering, which the talk describes as a subset of it, is configuring the surrounding tooling (the agent's commands, rules files, subagents, hooks and how they pass information) so that the right context shows up at the right time.

Idea 3: The "amount of lies" chart. A slide in the talk ranks information sources in a codebase by how often they mislead: function names are usually closest to the truth, comments are less trustworthy, and documentation is most likely to be out of date. The practical lesson: do not have an agent depend on stale docs. Instead let it read the actual code on demand and generate fresh, compact understanding.

Idea 4: Research, plan, implement. Break work into three phases with separate contexts:

• Research. The agent explores the codebase and produces a short document describing how the relevant parts work, with file names and line references. A human reviews it.
• Plan. Using the research, the agent writes a step by step implementation plan with the exact files, code changes and tests. A human reviews this too. Catching a wrong assumption here is far cheaper than catching it in code.
• Implement. The agent executes the plan, ideally in a fresh context window, updating progress as it goes.

Idea 5: Intentional compaction. Rather than letting the context grow until it is automatically summarised, you deliberately compress. Save the findings to a file and start a new session that reads only that file. You decide what matters.

Idea 6: Use subagents for context control. A subagent does a messy search in its own context and returns only a short answer. The parent context never sees the noise. The goal is not role-playing personas but isolating context.

Idea 7: Review the plan, not only the code. The leverage point shifts. One bad line of research can produce thousands of bad lines of code. Human attention is best spent on research and plan documents, which also keeps teammates aligned (mental alignment).

Idea 8: Do not outsource the thinking. The agent amplifies your understanding; it does not replace it. If you do not understand the problem, a plan from the agent will not rescue you. Match effort to difficulty: a one-line fix needs no plan, a cross-cutting change needs full research and planning.

Concepts glossary

Tools and links

Hands-On Exercise

Do this step by step

1. Choose a medium sized repo and a small real issue.
2. Start a fresh agent session. Ask for research only: how the relevant flow works, with file and line references. Save it to a markdown file.
3. Read it. Correct anything wrong. Delete anything irrelevant.
4. Start another fresh session with only that file. Ask for a plan: exact files, edits, tests and verification commands. Save and review it.
5. Start a third fresh session with the plan. Implement one phase at a time, running tests after each.
6. Keep the context short. When it grows, write progress to the plan file and restart.
7. Open a pull request whose description links the research and plan.

Acceptance criteria: three saved documents, tests passing, a diff a maintainer can review and a note on where the agent went wrong and how you corrected it.

Common mistakes

• One giant session that runs until quality collapses.
• Trusting documentation and comments over the code.
• Reading only the final diff and never the plan.
• Using the full workflow for trivial changes, or none of it for hard ones.
• Letting the agent decide scope without a human check.

How to get better at this

• Keep a rules file and edit it every time the agent repeats a mistake.
• Learn the codebase yourself. Agents multiply understanding, they do not replace it.
• Practise writing precise research prompts such as "describe how X works, cite file and line, do not propose changes".
• Review a plan critically: are the assumptions verified, is each step testable.
• Study how teams measure AI productivity honestly, including rework.

Interview questions

1. Why do coding agents struggle in large legacy codebases? Strong answer: limited context fills with noise, hidden conventions and stale docs mislead, and tasks span many files, so quality degrades.
2. Explain research, plan, implement and why the phases use separate contexts. Strong answer: each phase produces a small reviewed artifact; fresh contexts keep the model in its high quality range and let humans catch errors early.
3. What is context compaction and how is it intentional? Strong answer: deliberately distilling findings into a file and starting fresh, choosing what is kept, instead of relying on automatic summaries.
4. What is the point of subagents? Strong answer: to isolate noisy exploration in a separate context so the parent receives only a concise result.
5. Why are documentation and comments less reliable than code? Strong answer: they drift out of date because nothing enforces them, while code must work. Generate understanding from code on demand.
6. How would you measure whether AI coding helps your team? Strong answer: track cycle time, defect rate, review time and rework (code rewritten within weeks), not only lines produced.
7. Where should human review focus? Strong answer: on research and plans, where errors have the largest leverage, then on tests and the final diff.
8. When is a full workflow overkill? Strong answer: for small, well understood changes. Scale process to task difficulty.

Resources

• Advanced context engineering for coding agents
• 12-factor-agents
• Claude Code documentation
• Effective context engineering for AI agents
• AGENTS.md convention
• Simon Willison's blog on AI-assisted programming

T A L K   7   ────      7 / 7

Beating RL With Reflection: GEPA

Focus: Improve prompts and agents automatically by letting a model reflect on full traces, instead of training with reinforcement learning.

Why this talk matters

Once you can measure quality (talks 1 and 2) you can try to improve it automatically. GEPA (Genetic-Pareto reflective prompt evolution) is a method from researchers at UC Berkeley (including the Sky Computing Lab) and collaborators, with Lakshya Agrawal as lead author. Its paper reports that reflective prompt evolution can outperform reinforcement learning methods such as GRPO while using far fewer rollouts. The talk title, "Beating RL With Reflection", states the claim, and the talk extends it to "optimize anything" that can be represented as text.

This matters because reinforcement learning on a model is expensive and needs lots of examples, while prompt optimisation needs no weight updates and can work with a few dozen examples.

The big ideas explained

Idea 1: RL gives one number, language gives understanding. In reinforcement learning with a method like GRPO, the system runs many attempts (rollouts), gets a scalar reward for each and nudges the model toward higher reward. A single number says that an attempt was bad, not why. A model reading the full trace can see the exact mistake and write a fix in words.

Idea 2: Reflective prompt evolution. GEPA runs the system on a few examples and collects the traces: inputs, reasoning, tool calls, outputs and scores. A reflection model reads them, diagnoses problems in natural language and proposes a revised prompt. The new candidate is tested, and good ones are kept.

Idea 3: Pareto frontier. Instead of always keeping the single highest average candidate, GEPA keeps candidates that are the best on at least one example or objective. This preserves diversity: a prompt that solves hard cases the others miss stays in the pool and can be combined or improved. It avoids getting stuck in a local optimum.

Idea 4: Merge and mutate. New candidates come from mutation (reflection driven edits) and from merging lessons from different candidates, similar to genetic algorithms.

Idea 5: Sample efficiency. Because each rollout gives rich textual feedback, GEPA reportedly needs far fewer rollouts than RL to reach comparable or better scores. Check the paper for the exact multiples on each benchmark.

Idea 6: Rich feedback, not only scores. You can improve results by giving the optimiser more than a number. Return textual feedback from your metric: which test failed, which rubric item was missed, what the compiler said. The better the feedback, the better the reflection.

Idea 7: Optimize anything. The talk's second half generalises beyond prompts: any artifact that is text and can be scored can be evolved, for example code, configuration, agent instructions or even structured designs. The example mentioned in the talk of a thumbnail style prompt improving from 32 percent to 89 percent on a judged score illustrates how an LLM-as-judge metric plus reflection can raise quality quickly. Treat headline numbers as examples from the talk and verify on your own tasks.

Idea 8: Overfitting and judge risk. Optimising against a judge can exploit its quirks. Keep a held-out validation set and inspect final prompts by eye.

Concepts glossary

Tools and links

Hands-On Exercise

Do this step by step

1. Define the task, a scoring function and a train and validation split. Keep the validation set locked.
2. Measure the baseline on both sets.
3. Make the metric return feedback text as well as a score, such as "missed field date" or "answer not grounded in context".
4. Run GEPA with a small budget. Use a strong model for reflection and your target model for the task.
5. Inspect the evolved prompts. Read them. Look for rules that look like memorised answers.
6. Evaluate the best candidate on the validation set. Compare with baseline and with a simple hand-tuned prompt.
7. Record cost, number of rollouts and score gains. Keep the winning prompt in version control.

Acceptance criteria: a held-out improvement over baseline, a readable final prompt without leaked answers, and a recorded budget.

Common mistakes

• Optimising against a flawed metric and celebrating the number.
• No held-out set, so overfitting goes unnoticed.
• Returning only a score and wasting the main advantage of reflection.
• Letting optimisation run before doing error analysis. If the failures are data issues, a better prompt will not fix them.
• Treating the paper's headline numbers as guaranteed for your task.

How to get better at this

• Do talks 1 and 2 first: good optimisation needs a trustworthy metric.
• Read the GEPA paper and the DSPy documentation, then try other DSPy optimisers to compare.
• Learn when to optimise prompts and when to fine-tune. Prompt optimisation is cheap and transparent; fine-tuning helps when behaviour, format or latency need to change at the weight level.
• Build the habit of storing every prompt version with its score.

Interview questions

1. What is GEPA in one sentence? Strong answer: an optimiser that evolves prompts by having a model reflect on traces in natural language and keeping a Pareto set of diverse candidates.
2. Why can reflection be more sample efficient than RL? Strong answer: a trace carries far more information than a scalar reward, so each rollout teaches more.
3. What is the Pareto frontier used for here? Strong answer: preserving candidates that are best on some examples so diverse strengths are not lost and the search does not collapse onto one local optimum.
4. How do you stop optimisation overfitting? Strong answer: a locked validation set, regularly inspecting prompts, and checking improvements on fresh data.
5. What makes a good feedback signal for reflection? Strong answer: specific, actionable text that points to the failing step, rubric item or test.
6. When would you choose RL or fine-tuning over prompt optimisation? Strong answer: when you need behaviour that prompts cannot achieve, need lower latency or smaller models, or have lots of data and a stable task.
7. What risks come with LLM-as-judge metrics? Strong answer: bias, exploitable quirks and drift. Calibrate against human labels and use held-out human checks.
8. How does DSPy relate to GEPA? Strong answer: DSPy lets you write LLM programs as modules with signatures and optimise them; GEPA is one of its optimisers.

Resources

• GEPA paper
• GEPA repository
• DSPy documentation
• DSPy paper
• Hamel Husain on evals

How to be better: habits that compound

Watching talks does not make you better. Changing how you work does. The seven talks reduce to a handful of habits.

A ladder from beginner to senior

1. Beginner: can call a model API, use tools and build a simple chat or workflow.
2. Intermediate: can build an agent loop by hand, write structured outputs, and run a small eval set.
3. Advanced: runs error analysis, tracks reliability and cost, writes skills, and owns context and control flow.
4. Senior: designs the evaluation strategy for a team, knows when not to build an agent, optimises systems automatically with guardrails against overfitting, and sets honest expectations with stakeholders.

Fifteen things to do in the next 30 days

• Watch the talks in the recommended order and take five lines of notes each
• Build the error analysis spreadsheet for one real system
• Write 30 eval tasks and run each five times
• Write an "agent or workflow" checklist for three tasks
• Implement a bare agent loop with no framework
• Write two Agent Skills and test trigger accuracy
• Rebuild one small agent using the 12 factors
• Add a human approval step to something that has side effects
• Create a rules file for a repository you work on
• Run research, plan and implement on a real issue
• Optimise one prompt with GEPA or DSPy and verify on a held-out set
• Add tracing with Langfuse or Phoenix to one project
• Write a one page note on what your system should not be trusted with
• Share a short write-up or post about what you built
• Teach one talk to a friend in ten minutes

Capstone project: a reliable skill-powered agent with an eval loop

Combine everything into a single project you can show in interviews.

What to build. An agent that does one valuable job end to end, for example triaging inbound support emails, reviewing pull requests, or answering questions about internal documents.

Deliverables. A repository with a README that has an architecture diagram, an eval report with accuracy, reliability and cost, a failure-mode table with before and after counts, and a "do not use for" list.

Acceptance criteria. Replayable runs, a measured improvement after optimisation, human approval on risky actions, and a write-up under one page.

Interview bank: cross-cutting questions

1. Walk me through how you would decide whether to build an agent. Strong answer: use the four questions (complexity, value, doable parts, cost of error), start with a workflow, and justify autonomy with evidence.
2. How do you make an LLM system reliable? Strong answer: small focused steps, structured outputs, own the control flow, validate, retry with caps, human approval for risky actions, evals and tracing.
3. How do you know your change made things better? Strong answer: a fixed eval set with held-out data, repeated runs, cost tracked, error analysis to confirm the targeted failure mode dropped.
4. What is context engineering? Strong answer: designing what the model sees: instructions, tools, retrieved data, history and format, to maximise relevance and minimise noise.
5. When do you use a skill, an MCP server, a subagent or a tool? Strong answer: skill for procedure, MCP for access to systems, subagent for context isolation, tool for a single action.
6. How do you debug an agent that behaves strangely? Strong answer: read the trace, inspect exactly what was in context at the bad step, reproduce from the event log, fix the prompt or tool and add an eval.
7. How would you reduce cost without hurting quality? Strong answer: shorter context, caching, routing easy cases to smaller models, workflows instead of agents where possible, and checking on the accuracy and cost frontier.
8. How do you evaluate an LLM judge? Strong answer: compare to human labels on a sample, examine disagreements, refine the rubric and monitor drift.
9. Describe a time an AI system failed and how you handled it. Strong answer: describe the symptom, the labelled traces, the dominant failure mode, the fix and the eval that now guards it.
10. How would you introduce AI coding agents to a team? Strong answer: start with research and plan documents reviewed by humans, a shared rules file, measured cycle time and rework, and scope the process to task difficulty.
11. What are the security concerns of agents with tools and skills? Strong answer: prompt injection, excessive permissions, untrusted code. Use least privilege, sandboxing, approval steps and reviewed sources.
12. Where do you think agents are over-hyped and where under-appreciated? Strong answer: over-hyped for open ended unattended work with high error cost; under-appreciated for verifiable tasks such as coding and well scoped workflows with humans in the loop.

Tools master list

Reading list

• Building Effective Agents, Anthropic
• Effective context engineering for AI agents, Anthropic
• 12-factor-agents, Dex Horthy
• AI Agents That Matter, Kapoor and colleagues
• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, Agrawal and colleagues
• Machine Learning Yearning, Andrew Ng
• Your AI Product Needs Evals, Hamel Husain
• AI Engineering, Chip Huyen
• AI Snake Oil, Narayanan and Kapoor