1 of 21

Generating Laughter�Evaluating the Success of LLMs for Comedic Purposes

Erin Mikail Staples (she/her)

Sr. DX Engineer, Galileo.ai

2025 | 1

2 of 21

Generating Laughter�Evaluating the Success of LLMs for Comedic Purposes

Erin Mikail Staples (she/her)

Sr. DX Engineer, Galileo.ai

2025 | 2

3 of 21

Erin Mikail Staples (she/her)

developer experience engineer,�open source enthusiast, + ai-enhanced stand-up comedian

erinmikailstaples.com�erin@galileo.ai

2025 | 3

4 of 21

2025 | 4

5 of 21

2025 | 5

6 of 21

The roadmap.

Nondeterminism Isn’t a Bug, It’s the Bit

Understanding Metrics

Building your metrics stack

Business Case Examples

Startup Simulator 3000

2025 | 6

7 of 21

Nondeterminism Isn’t a Bug, It’s the Bit

2025 | 7

8 of 21

LLMs are unpredictable by design

But how do you test something you don’t expect to repeat?

2025 | 8

9 of 21

Keeping One Foot

on the Ground

2025 | 9

10 of 21

Standard AI Evaluation Metrics

✅ Accuracy: Is it truthful?

🔢 Token Usage: How many tokens did it burn?

🕓 Latency: How long did it take?

🛟 Safety: Did the model create harm?

2025 | 10

So these are the metrics most people start with when evaluating AI systems. Accuracy, completion rate, token usage, latency. They’re foundational—and they are helpful!

Accuracy and completion rate tell us if the model did the task. Token usage helps us keep our OpenAI bill under “oops.” And latency? It’s how long users stare awkwardly at a loading spinner.

BUT—these are mostly surface-level. They don’t tell us if the model actually understood the task. Or if it gave the user what they meant, not just what they said.

It’s kind of like judging a chef by how clean their apron is. I mean, sure, it tells you something... but it doesn’t tell you how the food tastes, whether they followed the recipe, or if they swapped sugar for salt.

�Users don’t care if your agent’s token usage is low if it gets the answer wrong. They don’t care if latency is fast if the response is unhelpful or confusing. They care about the outcome matching their expectation.

11 of 21

Serious Mode

Silly Mode

Pitch Decks

News API

Serious Formatter

Hacker News

Satire

Silly Formatter

2025 | 11

12 of 21

Traditional �AI Eval �Metrics

Agentic Experience

Metrics

✨�What really

matters

✨

2025 | 12

This diagram shows where we often start—and where we need to go.

On the left, we’ve got the classics: accuracy, token usage, BLEU scores, maybe some latency metrics. These are the comfort zone—easy to compute, easy to graph, and easy to misunderstand.

On the right, we’ve got what agentic systems actually require to be useful:

Did the agent understand what I meant?�
Did it make smart decisions along the way?�
Was it efficient with resources?�
And crucially—do I trust it not to hallucinate or mislead me?�

The overlap? That’s the happy accident zone. Sometimes a traditional metric like "task success" maps to a good user experience. But with agents, especially multi-step ones, “completed” doesn’t always mean “correct” or “sane.”

It’s like grading a student solely on whether they turned in the assignment—not what they turned in or how they got there. For agent systems, we have to go deeper.

13 of 21

Traditional �AI Eval �Metrics

Agentic Experience

Metrics

✨�What really

matters

✨

Custom Metrics (involve SME)

2025 | 13

This diagram shows where we often start—and where we need to go.

On the left, we’ve got the classics: accuracy, token usage, BLEU scores, maybe some latency metrics. These are the comfort zone—easy to compute, easy to graph, and easy to misunderstand.

On the right, we’ve got what agentic systems actually require to be useful:

Did the agent understand what I meant?�
Did it make smart decisions along the way?�
Was it efficient with resources?�
And crucially—do I trust it not to hallucinate or mislead me?�

The overlap? That’s the happy accident zone. Sometimes a traditional metric like "task success" maps to a good user experience. But with agents, especially multi-step ones, “completed” doesn’t always mean “correct” or “sane.”

It’s like grading a student solely on whether they turned in the assignment—not what they turned in or how they got there. For agent systems, we have to go deeper.

14 of 21

Beyond Comedy:

Custom metrics for

any industry.

2025 | 14

15 of 21

What should I measure?

Start with key metrics - Focus on metrics most relevant to your use case.

2025 | 15

16 of 21

What should I measure?

Establish baselines - Understand your current performance before making changes

2025 | 16

17 of 21

What should I measure?

Track trends over time - Monitor how metrics change as you iterate on your system

2025 | 17

18 of 21

What should I measure?

Combine multiple metrics - Look at related metrics together for a more complete picture

2025 | 18

19 of 21

What should I measure?

Set thresholds - Define acceptable ranges for critical metrics

2025 | 19

20 of 21

What should I measure?

Start with key metrics - Focus on metrics most relevant to your use case

Establish baselines - Understand your current performance before making changes

Track trends over time - Monitor how metrics change as you iterate on your system

Combine multiple metrics - Look at related metrics together for a more complete picture

Set thresholds - Define acceptable ranges for critical metrics

2025 | 20

21 of 21

Keep exploring!

Erin Mikail Staples (she/her)

erin@galileo.ai

https://git.new/Data-AI-Summit

2025 | 21