1 of 18

AMAN KHAN

Beyond Vibe Checks: Mastering Evals

2 of 18

A bit about me

I've been building AI products for the last ~10 years

- Evals for self driving

- Spotify Machine Learning Platform - billions of signals a day for personalization

- Arize - director of product, series C, investment from Microsoft, datadog. We work with spotify, reddit, uber, duolingo

I'm a product manager for the past almost 10 years building AI products. Here's a few of the places that I've worked.

I actually helped build out perception evaluation for self-driving cars at GM Cruise, uh, sort of early in my career. And that's basically helping understand, basically when we make changes to the car, what's the impact that that's having on the end-to-end system. So sort of think of this as, e-vals for self-driving.

I then worked at Spotify on our machine learning platform, which powered, basically all of the data that went into training, and then using that same data for inference time. So this is what we call like the future store product. This is actually using billions of records and billions of rows of data per day to be used for model training and inference for these models.

Fast forward to today, now at Arize, we're an AI observability and e-vals platform. We've been around for about five years, been at the company for a little over three. We just closed our Series C, with participation from Microsoft and Datadog, investing in this last round as well.

And what we do is we help companies just like the ones I used to work at, like Spotify and Cruise. In fact, Spotify is a customer of ours, help them understand how to get their models and their AI applications from development all the way through to production in a way that they can understand what the impact of their systems are, so they can actually evaluate those systems during development. And then take that all the way through to production and continuously improve their applications, with either ML or these large language model generative applications these days as well.

3 of 18

What we’ll cover

My 5 principles
What are evals and why they matter
Evaluate “AI trip planning” data

4 of 18

Who is this session for?

5 of 18

My Five Essential Skills

AI Fundamentals
Customer Obsession with Curiosity
Rapid Prototyping
Learning from Great AI Experiences
Mastering Evals and Observability

6 of 18

My Five Essential Skills

AI Fundamentals
Customer Obsession with Curiosity
Rapid Prototyping
Learning from Great AI Experiences
Mastering Evals and Observability

7 of 18

My Five Essential Skills

AI Fundamentals
Customer Obsession with Curiosity
Rapid Prototyping
Learning from Great AI Experiences
Mastering Evals and Observability

8 of 18

LLMs Hallucinate

Your job is to make sure they don’t embarrass you, your company or brand

9 of 18

What is an Eval?

10 of 18

What is an Eval?

11 of 18

Evaluating with vibes…

| We Make AI Work

12 of 18

… to Thrive

Coding

| We Make AI Work

13 of 18

14 of 18

15 of 18

Looking ahead

What is fundamentally shifting in the PM role?

16 of 18

Q&A !

17 of 18

Want more?

Lenny’s Newsletter

Arize Eval Hub

18 of 18

Thank You!

@_amankhan

/amanberkeley

aiproductplaybook.com