Building AI Agents for Enterprise Customer Support
PRODUCT TANK TO
12-AUG-2025
Lessons learned on aligning AI to user expectations
Today’s talk
Ada Demo
How Ada Works
Lessons Learned
Lesson 1
DETERMINISM VS. EXPRESSIVENESS
GEN AI
Attempts #1-4 and #6
Attempt #5
LLM planner
Plain old code
Ask first: do you need creativity or certainty?
DETERMINISM VS. EXPRESSIVENESS
Can a rule/SQL query answer the question?
Do you own enough labelled data to beat the rule?
❌
🚢 Ship it
✅
📊 Classic ML
✅
🤖Gen. AI
✅
Do you need to generate or interpret natural language, code, or multimedia?
❌
Revisit your problem statement…
❌
Lesson 2
CONVENIENCE VS. PRECISION
“Just give it the full API response!”
{
"VER": "3.14.7",
"DATA": {
"ORDERS": [
{
"ID": "OD123456789",
"STAT": "SHPPD",
"LAST_UPD": "31-07-25 14:03",
"EST_ETA": "03/08/25",
"SHIP_ADDR": {
"NM": "John Doe",
"ADDR1": "123 Any St",
"CITY": "Toronto",
"REG": "ON",
"ZIP": "M5V1B8",
"CTY": "CA"
},
"LINES": [
{
"SKU": "98765-BLU-256",
"DESC": "Deluxe Widget Blue 256 GB",
"QTY": 1,
"TRK": {
"CARR": "CNDPOST",
"NUM": "CN123987",
"EVT": "At sort ctr"
}
}
],
"META": { "PROMO": true, "LEGACY_FLAG": 7 }
}
],
"ERR": null
},
"DBG": "ok"
}
Full API response
Filtered, labelled outputs
✅
Thank you for reaching out! Your order with ID OD123456789 has been shipped. It was last updated on 31-07-2025 at 14:03. The estimated delivery date is 03-08-2025.
The order is being shipped to Melissa Smith at 123 Rory St, Toronto, ON, M5V1B8, Canada. The carrier is Canada Post, and the tracking number is CN123987. The current status is At sort center.
If you have any more questions or need further assistance, feel free to ask!
Thank you for reaching out about your order. Your order with ID 123456789 has been shipped. Here are the details:
- Order Status: Shipped
- Estimated Arrival Date: August 3, 2025
- Carrier: Canada Post
- Tracking Number: CN123987
You can use the tracking number to check the delivery status on the Canada Post website. If you have any further questions or need assistance, feel free to let me know!
Shortcuts kill precision — give AI high-signal context.
CONVENIENCE VS. PRECISION
More is not better.
Source: Does AI Actually Boost Developer Productivity? (100k Devs Study) � Yegor Denisov-Blanch, Stanford
Lesson 3
SCALABILITY VS. SPECIFICITY
Oh no, that’s super frustrating! Let’s crack this open and get you back in the game 🔐 Hang tight — I’m on it!
Hey, my account is locked and I can’t access my funds. What’s going on?
North American Crypto Co.
That’s exactly what I want — friendly and easy.
South Asian Bank
F@*& no! You are not putting that in front of my customers.
How do you standardize qualitative judgements…?
Evaluating Ada: A specific rubric with specific examples
Repeated for common inquiries across a subset of Ada customers�(Finance, Software, Retail)
You can’t scale “good” without defining it.
SCALABILITY VS. SPECIFICITY
GenAI product management
🌶️
The hardest part is (still) meeting user expectations.
TL;DR:
Summary
Trade-Off | Lesson | Example(s) |
Determinism vs. Expressiveness | Ask first: do you need creativity or certainty? |
|
Convenience vs. Precision | Shortcuts kill precision — give AI high-signal context. |
|
Scalability vs. Specificity | You can’t scale “good” without defining it. |
|
— | The hardest part is still meeting user expectations. | – |
Resources
AI Engineering �by Chip Huyen
Thank you.
linkedin.com/in/buildwithmaggie
chenmaggie.com