1 of 43

Agents building Agents_

Using Coding agents and SDD to build reliable AI agents

2 of 43

About me_

Alfonso Graziano 🇮🇹🍝

AI Tech Lead @ Nearform

—

2

Let’s connect :D

3 of 43

Everyone wants

AI Agents

—

3

4 of 43

How do we do that?_

AI

—

4

5 of 43

The Problems with Building AI Agents

...and how to solve them partially, with other agents!

—

5

—

5

—

5

6 of 43

AI Agents: a refresher_

—

6

—

6

7 of 43

Two classes of problems_

Bad performances on evals

Bad performances on live data

—

7

—

7

8 of 43

Bad performances on evals - the golden dataset_

Bad performances on evals

—

8

—

8

9 of 43

We have an hello world agent_

—

9

—

9

10 of 43

A naive evaluator_

—

10

—

10

11 of 43

Results are very poor_

—

11

—

11

12 of 43

Some failure modes_

No tools_

The system doesn't have the tools it requires to operate correctly, or the tools are wrong and/or contains bugs

—

12

Wrong system prompt_

The system prompt doesn't align with the rules which are represented in the Golden Dataset

No context retrieval_

The agent is not able to fetch the relevant context to answer correctly

13 of 43

Can an AI agent improve another agent autonomously?

—

13

—

13

—

13

14 of 43

Karpathy's Autoresearch shows that this is possible_

—

14

—

14

15 of 43

How autoresearch works_

—

15

—

15

16 of 43

So, I built auto-agent_

https://github.com/alfonsograziano/auto-agent

—

16

—

16

17 of 43

And it actually works!_

+10% on a

Production agent

—

17

—

17

18 of 43

How does it works?

—

18

—

18

—

18

19 of 43

The core idea_

—

19

20 of 43

The human in the loop_

—

20

21 of 43

Step 1

create a job_

—

21

—

21

22 of 43

Step 2 - run the loop_

* we define how many iterations we want to run

—

22

—

22

23 of 43

Step 2.1

create baseline data_

—

23

—

23

24 of 43

The baseline report_

—

24

—

24

25 of 43

Step 2.2 - running one iteration_

*The generated hypothesis is based �on MEMORY.md, other report files with failures, codebase investigation etc

—

25

—

25

26 of 43

Step 2.x - running every iteration_

—

26

—

26

27 of 43

Step 3 - generate a changelog and evaluate the results_

—

27

—

27

28 of 43

A real test on an already optimized agent_

Found edge cases�
Improved the system prompt�
Improved tools description�
Fixed tools logic

29 of 43

Fixing bad performances on live data_

Bad performances on live data

—

29

—

29

30 of 43

How do we fix that?_

—

30

—

30

31 of 43

Step 1: The user tests the system_

—

31

—

31

32 of 43

We collect �tracing informations_

—

32

—

32

33 of 43

Step 2 (a): The user gives a feedback_

—

33

—

33

34 of 43

Step 2 (b): Subject Matter Expert annotates the trace_

—

34

—

34

35 of 43

Step 3: Collect all the traces with feedback locally_

npm run fetch-traces-with-feedback.ts --from 2026-03-30 --to 2026-04-03 --limit 200

—

35

—

35

36 of 43

Step 4: Run the analysis_

—

36

—

36

37 of 43

Report example (1/2)_

—

37

—

37

38 of 43

Report example (2/2)_

—

38

—

38

39 of 43

Step 5: Prioritize and act on the feedback_

—

39

—

39

40 of 43

Continuous improvement_

Failure modes found during this investigation step become part of the golden dataset, and the evals suite is updated to spot regressions.

This report gets generated at least once per sprint.

In multiple cases a coding agent has been able to fix an entire suite of issues zero shot.

Complex cases might require the usage of auto-agent

—

40

41 of 43

Harness Engineering

All this is possible, thanks to

—

41

https://martinfowler.com/articles/harness-engineering.html�https://openai.com/index/harness-engineering/

42 of 43

Harness Engineering refers to the idea of building the environment around AI coding agents so they can work reliably: constraints, tests, feedback loops, and governance.

—

42

Spec-Driven Development�

A methodology which allows us to create specs which define our intention, constraints, functional requirements and an implementation plan for the agent.

Quality gates�

Unit tests on the codebase to catch regressions early, a CI/CD pipeline, regression evals, linting, security scanning and much more - to catch issues soon!

Context engineering�

Every repeated workflow becomes a skill. Important things go in rules files or CLAUDE.md files. Creating context becomes the key to continuous improvement.

Observability�

If we don't know what happens when we ship in production, we're blind. Proper observability in place allows us to quickly catch issues and debug properly.

43 of 43

Thank you!_