Evaluating LLMs & LLM Systems
January 2024
Overview
Topics we’ll cover
Why is evaluation important?
It’s never been easier to get started with ML…
Building demo != building production system
Why do things quickly fall apart?
A new, un-established technology + lots of moving parts
Errors in my LLM system
Building demo != building production system
Demo
Production
As we move towards production, it quickly becomes apparent that the landscape for error is vast
How an evaluation framework helps
Enable us to quantitatively measure how well our system or product is doing and detect any regressions
Ensure Performance
and Safety
Make Decisions and Ship Changes Faster
Identify Areas
for Improvement
A representative set of evals takes us a step towards reliably measuring system changes at scale - without evals, we are flying blind!
TL;DR - evaluation builds trust for both developers and users
AI Builder
Product Users
🤝
Why is LLM evaluation particularly challenging?
1. Lack of objective ground truth (and therefore reliable metrics)
https://www.turing.com/kb/generative-models-vs-discriminative-models-for-deep-learning �https://www.analyticsvidhya.com/blog/2021/07/deep-understanding-of-discriminative-and-generative-models-in-machine-learning/#h2_5/
Generative ML is different from discriminative ML
1. Lack of objective ground truth (and therefore reliable metrics)
Discriminative ML has clear, quantifiable metrics
Input
Label
Pred
muffin
chihuahua
muffin
chihuahua
muffin
chihuahua
chihuahua
chihuahua
accuracy = 75%
1. Lack of objective ground truth (and therefore reliable metrics)
Discriminative ML has clear, quantifiable metrics… Generative AI does not
Input
Label
Pred
muffin
chihuahua
muffin
chihuahua
muffin
chihuahua
chihuahua
chihuahua
accuracy = 75%
Input
Label
Pred
“Photo of a sheepdog”
“Dog that looks like a mop”
“A tangled mop”
“Picture of a mop head”
What metric ??
1. Lack of objective ground truth (and therefore reliable metrics)
What makes a piece of generated text “good”?
1. Lack of objective ground truth (and therefore reliable metrics)
It’s very subjective…
Truthful?
Accurate?
Ethical?
Safe?
Helpful?
Honest?
Creative?
Funny?
Formatting?
Relevance?
Concise?
1. Lack of objective ground truth (and therefore reliable metrics)
Often immeasurable…
Truthful?
Accurate?
Ethical?
Safe?
Helpful?
Honest?
Creative?
Funny?
Formatting?
Relevance?
Concise?
Cool, so now how do we measure this??
2. Diversity of possible tasks
LLMs can be used in many ways
2. Diversity of possible tasks
accuracy = 92%
accuracy = ???
Scope of LLM capability >> available eval benchmarks
3. Our current methods aren’t mature enough
Evals are a nascent field - more of an art than a science
Ex 1. Slight changes in prompt format result in wildly different eval scores
Ex 2. Tipping a LLM $300k leads to better response:
So, what options do we have for evaluating LLMs?
Evaluation methods
Four ways LLMs are evaluated today
1. Public Benchmarks
2. Functional Correctness
3. Human Evaluation
4. Model-based Evaluation
Evaluation methods
Four ways LLMs are evaluated today
1. Public Benchmarks
2. Functional Correctness
3. Human Evaluation
4. Model-based Evaluation
1. Public Benchmarks
There are lots of them, of varying quality, for varying tasks…
46 Popular Benchmarks
MMLU has 57 Tasks
Benchmarks often have subtasks
1. Public Benchmarks
Task-specific benchmarks are often grouped into collections to form “test-suites” & “leaderboards”
Popular Collections/Leaderboards:
Language Understanding | Commonsense Reasoning | Reading Comprehension | Programming | World Knowledge | Symbolic Problem Solving |
HellaSwag LAMBADA Winograd Schema Challenge Winogrande Big-bench: language identification Big-bench: conceptual combinations Big-bench: conlang translation | Big-bench: Strategy QA Big-bench: Strange Stories Big-bench: Novel Concepts COPA OpenBook QA PIQA | SQuaD BoolQ Big-bench: Understanding fables Pubmed QA Labeled | HumanEval: code generation | MMLU Jeopardy Big-bench: wikidata ARC easy ARC challenge Big-bench: misconceptions | Math QA LogiQA Simple arithmetic with spaces Big-bench: elementary math QA Big-bench: dyck languages Big-bench: algorithms Big-bench: logical deduction |
1. Public Benchmarks
Ex: MMLU - Measuring Massive Multitask Language Understanding [2020]
About
1. Public Benchmarks
Ex: GSM8K - Grade School Math 8k [2021]
About
1. Public Benchmarks
Issues with public benchmarks
1. Public Benchmarks
Case Study on MMLU: Are we measuring what we think we are?
MMLU scores on Open LLM Leaderboard didn’t match those reported in the LLaMA paper??
1. Public Benchmarks
Case Study on MMLU: Three different implementations of the same benchmark (Original, HELM, AI Harness)
What are the differences?
1. Public Benchmarks
Case Study on MMLU: Three different implementations of the same benchmark (Original, HELM, AI Harness)
Original
1. Public Benchmarks
Case Study on MMLU: Three different implementations of the same benchmark (Original, HELM, AI Harness)
HELM
1. Public Benchmarks
Case Study on MMLU: Three different implementations of the same benchmark (Original, HELM, AI Harness)
Eleuther AI Harness
1. Public Benchmarks
Evaluation results are strongly tied to their implementation details. This shows importance of standard evals.
Evaluation methods
Four ways LLMs are evaluated today
1. Public Benchmarks
2. Functional Correctness
3. Human Evaluation
4. Model-based Evaluation
2. Functional Correctness
Using functional tests to explicitly determine if output of model solves the task at hand
Prompt
Unit tests
Predicted solution
HumanEval Benchmark
Example Leaderboards:
2. Functional Correctness
How can this apply outside of code generation?
Evaluation methods
Four ways LLMs are evaluated today
1. Public Benchmarks
2. Functional Correctness
3. Human Evaluation
4. Model-based Evaluation
3. Human Evaluation
Vibe check!
3. Human Evaluation
Manual review + annotation
Common Human Assessment Criteria
3. Human Evaluation
Live, continuous human preference evaluation
Evaluation methods
Four ways LLMs are evaluated today
1. Public Benchmarks
2. Functional Correctness
3. Human Evaluation
4. Model-based Evaluation
4. Model-based Evaluation
“LLM as a Judge”
“Strong LLM judges like GPT-4 achieve over 80% agreement rate - on par with that of human experts.”
Example Prompt
4. Model-based Evaluation
“LLM as a Judge”: alternatives to GPT4
Open-source LLMs that are fine-tuned to be evaluators of other models.
Demonstrate evaluation capability on par with GPT4.
Can be further fine-tuned!
4. Model-based Evaluation
Limitations of “LLM as a Judge”
https://twitter.com/aparnadhinak/status/1748368364395721128/photo/1
https://eugeneyan.com/writing/llm-patterns/#evals-to-measure-performance
If LLMs need to be evaluated because they’re unreliable… why should we trust an LLM to do the evaluation???
Known Biases
Other Issues
Comparison of methods
| Benefits | Limitations |
1. Public Benchmarks |
|
|
Comparison of methods
| Benefits | Limitations |
1. Public Benchmarks |
|
|
2. Functional Correctness |
|
|
Comparison of methods
| Benefits | Limitations |
1. Public Benchmarks |
|
|
2. Functional Correctness |
|
|
3. Human Evaluation |
|
|
Comparison of methods
| Benefits | Limitations |
1. Public Benchmarks |
|
|
2. Functional Correctness |
|
|
3. Human Evaluation |
|
|
4. Model-based Evaluation |
|
|
Evaluating LLM systems on your data
Evaluation Driven Development (EDD)
Curate a task-specific eval dataset and use it to guide prompt engineering and design choices while developing
Develop
Auto Evaluation Framework
Approve?
Curate & Validate Eval Dataset
Deploy
Feedback
No
Yes
Invest in building an evaluation framework
What makes a good evaluation framework?
Case Study: RAG for Factoid QA Over Internal Documents
1. Evaluation data that is representative of production data
Generate synthetic data to start quickly, improve the dataset over time with feedback
Given the following context, write a factoid question that is answerable from the context alone, and then provide the correct answer.
Output Format:
```
{
“question”: <question>,
“answer”: <answer>
}
```
Context:
<context>
Output:
Knowledge Base
Context Chunk (from knowledge base) | “We offer a comprehensive vacation policy that allows full-time employees to accrue 25 days of paid vacation per year, accruing monthly, starting after the completion of the probationary period of 3 months.” |
Question (generated) | How long does it take to start accruing paid vacation days? |
Answer (generated) | 3 months |
Prompt
Synthetic Eval Data
2. Evaluation metrics that are correlated with outcomes
Select a few meaningful metrics that provide signal on the efficacy of your system
Retriever: Precision @ K
Generator: Answer Correctness
How many items in the top-k ranked search results contain the reference answer?
Is the generated response consistent with the reference answer?
(b)
(a)
“LLM as a Judge”
“LLM as a Judge”
3. Evaluation method that is fast, automated, and reliable
Method should be representative of human evaluation, but allow for quick development iterations
LLM
Developer
Auto-Eval
“LLM as Judge”
Manual Human
Evaluation/Validation
Fast & Automated
Reliable
Thank You!