Use AI�and check its work
Session 5 of 8 · AI-Assisted Development
© The Build Fellowship 2024
You've made the codebase faster. Now we let AI in.
psychology
We are past whether we should use AI and now to how we use AI.
Today we generate code with AI, then review it as critically as we would an engineer's pull request.
© The Build Fellowship 2024
Agenda
AI in Engineering
© The Build Fellowship 2024
© The Build Fellowship 2024
Three modes of working with AI
keyboard
Autocomplete
Copilot, Cursor, your IDE. The AI suggests the next few lines as you type. Low-friction, high-volume. Great for boilerplate. Easy to over-trust because it feels harmless.
forum
Chat
Claude, ChatGPT, Cursor's chat mode. You ask, it generates a chunk of code or explanation. Higher leverage per interaction, but the quality of the prompt matters a lot more. This is what we use today.
smart_toy
Agents
Autonomous execution. You define a high-level goal, and the AI plans, executes, and debugs. Extremely high leverage, but requires careful monitoring and verification as it acts independently.
What good prompting actually looks like
The most common mistake is treating the AI like Google.
The second most common mistake is treating it like a senior engineer who already knows your codebase.
Treat it like a new contractor who can read everything you show it.
1
Give context
Paste the existing code, and mention the libraries or design conventions you follow.
2
Be specific
Explicit directions like "rewrite this to use early returns" always beat generic requests like "make it better."
3
Ask for validatable output
Request outputs that are easy to verify programmatically or visually, such as tests, types, or before/after diffs.
4
Iterate the prompt, not the code
When the AI's output is wrong, the prompt itself is usually wrong. Refine your constraints instead of manually editing the output.
© The Build Fellowship 2024
The cost
© The Build Fellowship 2024
© The Build Fellowship 2024
Cognitive offloading
The thing to watch for
Offloading is when you hand the thinking to a tool and stop building the mental model yourself.
Understanding is the ultimate bottleneck in software. When you offload comprehension, the true cost comes later.
When it works fine
A calculator does it for arithmetic, GPS does it for navigation. These tools handle execution where deep conceptual mastery isn't the primary goal.
With AI writing code, it's different
It happens fast, it happens invisibly, and the output looks like you wrote it yourself.
You read code that works, you accept it, and you never form the understanding that writing it yourself would have forced.
Why this is a real problem, not a vibe
Typing was never the bottleneck in software. Understanding is. When you offload the understanding, the bill comes due later.
Individual Risks
• Blind Spots: You cannot debug, review, or extend code you never understood. The first incident finds the gap.
• False Security: Over-trust ships subtle bugs: the code reads as correct, so nobody looks hard.
Team & Long-Term Impact
• Skill Atrophy: The reps you skip today are the judgment you lack next year.
• Compounding Gaps: On a team, the gaps compound. Nobody owns the parts the AI wrote and nobody read.
© The Build Fellowship 2024
One question tells
you if you
offloaded
© The Build Fellowship 2024
The Golden Question
"Could you re-derive this, or explain it to a teammate, without the AI in front of you?"
The Diagnostic
• If yes: You offloaded the typing, which is fine.
• If no: You offloaded the comprehension, which is the trap.
Code you do not understand is a liability you have not noticed yet.
Build comprehension into the system prompt
The system prompt is the highest-leverage place to fight offloading, because it shapes every answer in the session. Instead of asking the AI to just emit code, instruct it to keep you in the loop. You will write your own version of this in the activity.
Core Instructions
• Explain Reasoning: Not just the result.
• Surface Assumptions: And the tradeoffs it accepted.
• Offer Alternatives: Say why it picked this one.
Interactive Verification
• Ask Clarifying Questions: Have the AI ask you instead of guessing.
• Quiz the User: Prompt you to explain the output so you have to articulate why it works.
© The Build Fellowship 2024
© The Build Fellowship 2024
Habits that keep the understanding yours
The system prompt sets the AI's behavior. These habits set yours.
01 Read and understand before you accept. No exceptions for code that "looks right."
02 Rewrite the AI's explanation in your own words. If you cannot, you do not understand it yet.
03 Review AI code like an engineer's PR: assume nothing, check everything.
04 Write or understand the test first, so you can judge the code against intent.
05 Modify the output without AI to prove it is yours. If you cannot change it, you do not own it.
After the AI answers, run the loop.
Three questions, every time the AI produces something you intend to keep. This is the loop you will practise in today's activity.
01 Why does this work?
Say it out loud or write it down.
02 What breaks if an assumption changes?
For example, an input is empty, a client times out, or the schema shifts.
03 What tradeoff did we accept?
Consider why we did it this way instead of the alternative.
© The Build Fellowship 2024
© The Build Fellowship 2024
What bad prompting produces
THE LAZY APPROACH
"make this function better."
Lacks critical context. Produces a generic, low-quality response that often misses the actual goal.
THE CAREFUL PROMPT
Visibly superior quality. Guided directly by engineering intent.
The ultimate takeaway: Prompts are first-class code. Treat them with the same precision and care.
Refactor�a real function.
© The Build Fellowship 2024
© The Build Fellowship 2024
This is the target
Open apps/api/src/routes/reports.ts
The Complex Monolith
~130 lines doing ten things at once
•Queries tasks and hydrates assignees
•Computes rates and applies timezones
•Buckets results into a CSV report
•Dispatches final exit notification
The Refactoring Plan
Deconstruct safely using AI
1. Split It Apart Ask AI to break the function into single-responsibility, modular helpers.
2. Verify Integrity Run the existing test suite immediately to confirm nothing broke during the split.
Refactor `reports.ts` with AI help
© The Build Fellowship 2024
1. Preparation
Follow along in your own clone.
System Prompt
Set the comprehension-preserving prompt from active-comprehension-prompt.md
Template
Use the template in refactor-prompt.md
2. AI Refactor
Paste Code
Paste the contents of reports.ts into the prompt.
The Goal
Ask for a refactor that splits the function into smaller, modular helpers.
Apply
Apply the AI's suggestions to your file.
3. Verification
Run Tests
Verify with test suite. If failed: fix regression & update prompt.
Comprehension Check
Name each helper and its function from memory without re-reading.
Record
Document your results in your logbook.
Generate tests�for code that has none.
© The Build Fellowship 2024
© The Build Fellowship 2024
Zero tests
Open�auth.service.ts
This file is absolutely critical for security, yet it currently has zero coverage. Today you will change that with AI help.
Why test this file?
It is responsible for critical authentication mechanisms: hashing passwords, signing JWTs, and validating tokens.
The Prompt Template Playbook
Feed the AI the core components to get robust, accurate tests:
Write tests for auth.service.ts with AI
© The Build Fellowship 2024
01 / SETUP & RUN
We do this together, system prompt still on. Use the template prompt:
test-gen-prompt.md
Generate and run the tests.
02 / ITERATE & FIX
They probably won't pass on the first try. AI often:
Fix what's broken and log it.
03 / VERIFY
Comprehension Check
Change an assertion to something you know is wrong.
Predict the failure before running. If the actual failure surprises you, re-evaluate!
Now grade�its work.
© The Build Fellowship 2024
© The Build Fellowship 2024
Grade the AI output against four axes.
01 / CORRECTNESS
Does it pass the existing tests?
Does the behavior match the spec?
02 / MAINTAINABILITY
Would a teammate be able to read and modify this six months from now?
03 / COVERAGE
Does the AI handle edge cases, or only the happy path?
04 / IDIOM
Does it match the conventions of the surrounding code?
© The Build Fellowship 2024
Review your AI output
Together, for each piece of AI output we applied (refactor + tests), we grade it on all four rubric axes.
Capture *specific* examples: "the AI didn't handle the empty-tasks case" beats "coverage was okay."
👉 Record your grades and justifications in your logbook as we go.
When AI tends to go awry
A pattern of failures you'll see across many tools.
01 / HALLUCINATIONS
Confident hallucinations of behaviour that doesn't
02 / EDGE CASES
Edge cases silently ignored
03 / TYPING
"Plausibly typed" but actually wrong TypeScript: types compile but lie.
04 / MISALIGNMENT
Code that solves a slightly different problem than the one you asked about.
© The Build Fellowship 2024
Group�discussion
© The Build Fellowship 2024
© The Build Fellowship 2024
Where did AI help, and where would blind acceptance have bitten you?
01 / SHARE EXPERIENCE
Around the room, share one place where AI saved you time, and one place where accepting output blindly would have caused problems
02 / SPOT THE GAP
Name a specific moment from today's comprehension checks where you initially thought you understood the code, but actually did not.
03 / LOG FOR POLICY
Record these near-misses in your logbook. These real-world learnings directly shape the final engineering policy we establish in Session 7.
© The Build Fellowship 2024
What this means for you
Engineers who use AI well are pulling ahead
01 / DIRECT THE TOOL
Hiring managers look for engineers who can direct AI without being directed by it. The core skill is judgment: what to ask for, what to accept, and what to discard.
02 / PROTECT UNDERSTANDING
Judgment is built entirely on deep understanding. Blindly offloading tasks to AI erodes comprehension, taking away your ability to evaluate the output critically.
03 / MULTIPLY VELOCITY
The engineers pulling ahead use AI to move faster and keep their mental model intact. That is the compounding practice that separates high performers.
© The Build Fellowship 2024
SESSION 06 / METRICS
Experimentation and measuring impact
You have a measurably faster pipeline, a better developer experience, and AI-assisted improvements. Now, prove it with metrics.
01 / BASELINES
Revisit your Session 1 baselines and compare them directly against your deployment pipeline performance today.
02 / DORA METRICS
Learn the key DORA metrics—deployment frequency, lead time, MTTR, and change failure rate—that engineering leaders care about.
03 / ROLLOUT PLAN
Design a comprehensive rollout plan for successfully introducing your optimized engineering changes to a broader development team.
Questions?
© The Build Fellowship 2024