Evaluation of�Information Access Systems �in the Generative Era��Negar Arabzadeh�University of Waterloo��University of California, Berkeley

Summer 2024

Narabzad@uwaterloo.ca

https://www.negara.me/

Introduction

2

Recommender Systems

Social Media

Search Engines

Generative Models

Introduction

3

Introduction

4

Introduction

5

This Photo by Unknown Author is licensed under CC BY-NC

…

Introduction

6

This Photo by Unknown Author is licensed under CC BY-NC

….

….

Evaluation of Information Access Systems

• Rapid advancement of technologies 🡪 Evaluation become more challenging
• Developing new metrics aligned with unique characteristics of new systems

​

7

Evaluation of Information Access Systems

• Rapid advancement of technologies 🡪 Evaluation become more challenging
• Developing new metrics aligned with unique characteristics of new systems
• Evaluation requires data

​

​

​

​

​

• Limitations of extensive evaluations 🡪 Biased or incomplete assessments

​

8

Introduction

9

How do we provide environments where correct information is:

Available, Identifiable, and accessible?

Outline

10

Evaluation of understanding grounded language

​

​

Evaluation of generative IR systems

​

​

Evaluation of LLM-based applications

​

​

Evaluation of LLMs robustness in open-domain QA

​

​

Future directions

Evaluation of Grounded Language Understanding

11

Self-Evaluation through Conversations

• A system predicts its own failure and use it as a feedback:
• Conversation with user
• Asking the user to clarify their intent
• Announcing the system’s failure

12

Self-Evaluation through Conversations

• Neurips 2022 IGLU challenge: Interactive Grounded Language Understanding in a Collaborative Environment
• Build interactive agents
• In a Minecraft environment 
• learning to solve a task while provided with grounded natural language instructions 
• in a collaborative environment.

​

13

https://www.iglu-contest.net/

https://www.aicrowd.com/challenges/neurips-2022-iglu-challenge

​

Self-Evaluation through Conversations

• Asking clarifying questions is essential for developing human-like dialogue systems.
• “When” and “What” to ask as clarifying questions.

​

​

​

​

14

https://www.iglu-contest.net/

https://www.aicrowd.com/challenges/neurips-2022-iglu-challenge

​

Self-Evaluation through Conversations

• Asking clarifying questions is essential for developing human-like dialogue systems.
• “When” and “What” to ask as clarifying questions.
• Collecting a dataset
• Providing data collection tool
• Providing the training environment
• Designing the evaluation methodology

​

​

​

​

15

Dataset: https://github.com/microsoft/iglu-datasets

Data Collection Tool: https://github.com/iglu-contest/dataset-collection-and-evaluation

Online vs offline evaluation

• Exploring human preferences with automatic offline metrics.
• Collecting human explanations on why they prefer a certain agent.

​

​

​

​

​

​

16

Greenlands: https://github.com/microsoft/greenlands

Online vs offline evaluation

• Exploring human preferences with automatic offline metrics.
• Collecting human explanations on why they prefer a certain agent.
• Findings:

​

​

​

​

​

​

17

• Human preferences do not align with automatic offline evaluation metric e.g., F1.

Greenlands: https://github.com/microsoft/greenlands

Online vs offline evaluation

• Exploring human preferences with automatic offline metrics.
• Collecting human explanations on why they prefer a certain agent.
• Findings:
• Human preferences do not align with automatic offline evaluation metric e.g., F1.
• Human criteria are not necessarily captured in offline evaluation metrics.

​

​

​

​

​

​

18

Offline assessments must be aligned with human preferences, specifically on more complex and less well-defined tasks.

Self-Evaluation through Conversations

• Asking clarifying questions is essential for developing human-like dialogue systems.
• “When” and “What” to ask as clarifying questions.
• Collecting a dataset
• Providing data collection tool
• Providing the training environment
• Designing the evaluation methodology

​

​

​

​

19

Evaluation of Generative Information Retrieval

20

From Traditional to Generative Information Retrieval

21

Differences from traditional IR evaluation:

1. Non-deterministic results
2. No more “ten blue links”
3. No more traditional browsing models
4. No more traditional evaluation metrics i.e., nDCG

• Evaluation Anchor: Based on human implicit or explicit signals e.g., judged relevant documents.
• Reward Mechanism: Based the rank position of relevant documents retrieved by the ranker.

LLM for Relevance Judgements (LLM-as-a-Judge?)

• Automatic Evaluation of GenIR and RAG systems
• Using LLM for relevance judgements in IR

​

​

22

LLM for Relevance Judgements (LLM-as-a-Judge?)

• LLM vs. Human Judgments:
• High correlation between ranking of different retrieval systems when evaluating with traditional ranking metrics e.g., nDCG@10 with human judgements vs LLM judgements

23

TREC 2021 Deep Learning Data

LLM for Relevance Judgements (LLM-as-a-Judge?)

24

LLM for Relevance Judgements (LLM-as-a-Judge?)

Findings:

25

• Traditional relevance assessment for offline evaluation is no longer needed. LLMs are better than non-expert humans at judging basic relevance.

LLM for Relevance Judgements (LLM-as-a-Judge?)

​

​

26

LLM for Relevance Judgements (LLM-as-a-Judge?)

​

​

27

LLM for Relevance Judgements (LLM-as-a-Judge?)

​

​

28

Paul Héroult Charles Hall

Human judgments once rare and costly, can now be supplemented by LLMs, dramatically lowering costs and unlocking new evaluation opportunities.

GenIR Evaluation

• Before, human judgements were too expensive. Now that we have cheap LLM judgments, what can we do with them?
• Nugget-based evaluation

​

​

29

GenIR Evaluation

• Before, human judgements were too expensive. Now that we have cheap LLM judgments, what can we do with them?
• Nugget-based evaluation
• Pairwise preference judgments

​

30

GenIR Evaluation

• Before, human judgements were too expensive. Now that we have cheap LLM judgments, what can we do with them?
• Nugget-based evaluation
• Preference Judgments
• Binary/graded relevance assessment

​

31

• Relevant/ Non-relevant
• ​

Non-relevant

Highly relevant

GenIR Evaluation

• Before, human judgements were too expensive. Now that we have cheap LLM judgments, what can we do with them?
• Nugget-based evaluation
• Preference Judgments
• Binary/graded relevance assessment
• Similarity to sparsely labeled data

​

32

GenIR Evaluation

• Before, human judgements were too expensive. Now that we have cheap LLM judgments, what can we evaluations that were not practical before.

​

​

​

• We still need human feedback in information seeking systems, but what is the optimal role of human in the evaluation loop?
• The alignment between LLMs and real users needs thorough validation.
• Humans provide the objective function to optimize.

​

33

Gold is still Precious!

Evaluation through similarity to sparsely labeled data

• Challenges in generative-based tasks

Impractical to reassess the generated results

​

34

Evaluation through similarity to sparsely labeled data

• Challenges in generative-based tasks

Impractical to reassess the generated results

​

35

What evaluation strategies are being used in other generative based tasks i.e., Image generation?

Comparing generated output with good examples!

Evaluation through similarity to sparsely labeled data

• Challenges in generative-based tasks

Impractical to reassess the generated results

​

36

What evaluation strategies are being used in other generative based tasks i.e., Image generation?

Comparing generated output with good examples!

Sparse labels

Evaluation through similarity to sparsely labeled data

37

• Advantage 1: Unlike traditional retrieval assessment, there is no need to exactly retrieve the good examples to gain reward.

Target

document

Gain = 0

Traditional Retrieval Evaluation

Evaluation through similarity to sparsely labeled data

38

• Advantage 2: Allowing to have evaluation of both retrieval and generated answers in the same space.

Evaluation through similarity to sparsely labeled data

39

Evaluation through similarity to sparsely labeled data

40

Fréchet Distance

41

Fréchet Distance

42

Fréchet Distance for TTI

• Evaluation of Text-to-Image (TTI) generative models with Fréchet distance:

43

CAPTION

Text to Image generative model

Generated Images

Embedding Model

0.6 0.8 … ...

0.1 0.3 … …

FRÉCHET

DISTANCE

Fréchet Distance for IR

• Evaluation of IR systems with Fréchet distance:

44

• Relevant Judged

documents

Retrieved or generated

documents

Embedding Model

0.6 0.7 … ...

0.5 0.1 … …

FRÉCHET

DISTANCE

Fréchet Distance for evaluation of GenIR

45

Experimental Setup

• Classic Information retrieval test collection of items e.g., documents.
• Query sets with sparsely labeled documents
• A set of different retrievers/generative models to assess

​

46

Fréchet distance Experiments

Can Fréchet Distance effectively evaluate IR systems with sparse labels?

• Experiment :
• MRR@10 vs. FD@10 on ordering rankers

​

• Findings:
• FD can effectively pick out the better retriever, particularly when there is a significant difference between their performances.

​

47

Fréchet distance Experiments

Can the Fréchet Distance effectively evaluate IR systems when the retrieved results are not labelled?

​

​

48

​

​

​

​

Unjudged

​

Judged

​

Unjudged

​

​

Unjudged

​

​

…

Initial retrieved list

Traditional IR evaluation metric like nDCG assess the performance based on where the relevant judged document is placed.

Fréchet distance Experiments

Can the Fréchet Distance effectively evaluate IR systems when the retrieved results are not labelled?

​

​

49

​

​

​

​

Can we evaluate this list?

Unjudged

​

Judged

​

Unjudged

​

​

Unjudged

​

​

…

Unjudged

​

Unjudged

​

Unjudged

​

​

Unjudged

​

Initial retrieved list

Traditional IR evaluation metric like nDCG assess the performance based on where the relevant judged document is placed.

Fréchet distance Experiments

​

​

50

• Fréchet Distance can assess unlabeled data.
• When ranking a set of retrievers it shows statistically significant ranked based correlation on ordering rankers based on traditional IR metric.
• In contrast, traditional IR metrics would be unable to provide any insights without retrieving the labeled documents.

​

​

Can the Fréchet Distance effectively evaluate IR systems when the retrieved results are not labelled?

Evaluation of GenIR

51

Evaluation of GenIR with Retrieval Benchmark

52

Experimental Setup

• LLMS used under experiment:
• LLama2-7b-chat
• LLama2-13b-chat
• Gpt-3.5-turbo
• Gpt-4

53

Do goldfish grow?

Validation through Cross-Grade Relevance Similarities

Experiment:

• Selecting one “target response” for each query.
• Measuring similarity between representation of the target response and documents with different grade of relevance.
• ​

54

TREC DL 2019

Validation through Cross-Grade Relevance Similarities

Experiment:

• Selecting one “target response” for each query.
• Measuring similarity between representation of the target response and documents with different grade of relevance.
• ​

Findings:

• Validating usage of similarity for evaluation of QA task.

​

55

TREC DL 2019

• Robust w.r.t the choice of the document embedding.
• Passages with lower relevance levels show lower similarity scores, reflecting their lessor relevance to the information need.

​

Retrieved vs. Generated answers

​

Findings:

• We can assess both generated and retrieved models in a uniform context.
• Statistically significant Kendall’s 𝜏 correlation with ndcg@10.

56

ndcg@10

Challenges in Evaluation of GenIR

57

What if we have no “good” example?

Assessing Responses without Relevance Judgments

• Challenge: No human judgments are available.
• Solution: Using top-passage returned by different retrieval methods as an evaluation anchor and compare it with the generated answers.

​

58

Assessing Responses without Relevance Judgments

​

• Findings:
• Regardless of the choice of the retriever cheap or expensive, the top retrieved passage consistently emerges as a strong indicator of relevance.
• Relative performance of the models remains nearly unchanged when using different retrieval methods.

​

59

Evaluation of GenIR

60

Evaluation of LLM-based Applications

61

Evaluation of LLM-based Applications

62

Evaluation of LLM-based Applications

63

Turn on the lamp

Brainstorming on the paper title

Evaluation of LLM-based Applications

64

Give me a recipe with mushroom and chicken

Turn on the lamp

Evaluation of LLM-based Applications

65

Give me a recipe with mushroom and chicken

Turn on the lamp

Evaluation of LLM-based Applications

• AgentEval: Assessing the Task Utility of LLM-powered Applications for Their End-Users

​

​

​

​

​

​

​

​

​

​

• Math problem as an example to go beyond the success/failure of the method.

​

​

​

66

https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval/

Evaluation of LLM-based Applications

• AgentEval: Assessing the Task Utility of LLM-powered Applications for Their End-Users

​

​

​

​

​

​

​

​

​

​

• Math problem as an example to go beyond the success/failure of the method.

​

​

​

67

https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval/

Evaluation of LLM-based Applications

68

https://github.com/microsoft/autogen/blob/main/notebook/agenteval_cq_math.ipynb

Clarity Efficiency … completeness

Error

Analysis

Evaluation of LLM-based Applications

• AgentEval: Assessing the Task Utility of LLM-powered Applications for Their End-Users

​

​

​

​

​

​

​

​

​

​

​

​

69

How to validate the

Llm-based evaluation?

https://microsoft.github.io/autogen/blog/2023/11/20/AgentEval/

Evaluation of LLM-based Applications

70

• Integrated in Autogen Public library with more than 29k stars.
• Adopted internally by Microsoft product teams.

AgentEval Results

Findings:

• Validation on the Evaluation: Sanity check on if AgentEval is able to distinguish between failed and successful cases.
• Not all failed cases are the same.

71

AgentEval Results

Findings on quantification robustness:

• Studying reliability and reproducibility of assessments over 50 repeats.
• Some criteria lack clear distinguishability between failed and successful cases.
• The narrower the distribution, the more robust the criteria.

​

72

Robustness

73

Hallucination in Generative models

​

​

​

​

​

​

74

RAG → GARAGE

Retrieval Augmented Generation

Generate an Answer, Retrieve, Augment, Generate w/ Evidence

Hallucination in Generative models

​

​

​

​

​

75

Self-detecting hallucination:

• The ability for LLMs to self-detect hallucinations by confirming its generated responses against an external corpus. containing known correct answers.

​

Hallucination in Generative models

​

​

​

​

​

76

• Strengths:
• This approach is straightforward – minimal prompt engineering.

​

• Challenges:
• The reliance on retrieval.

Hallucination in Generative models

Stepped classification of QA pairs:

• LLM can correctly detect its own hallucinations in a majority of cases (an accuracy of over 80%), with the help of retrieval methods.

​

77

Hallucination in Generative models

78

• Strengths:
• Ensuring that both the generated and retrieved answers are directly related to the question.

​

• Challenges:
• Misclassifying answers due to excessive detail in the retrieved passages.
• Struggling with false negatives, especially when the retrieved evidence is too detailed or indirect.

Fact-checking in Generative models

​

​

• False positive rate are much more than False negatives.

79

Fact-based self-detecting hallucination:

Fact-checking in Generative models

​

​

• Strengths:
• More granular verification of generated content.
• Identifying hallucinations at a finer level.
• By validating individual statements, it can correct specific parts of an answer while maintaining overall coherence.

​

80

Fact-based self-detecting hallucination:

• Challenges:
• Over-generating factual statements, leading to unnecessary checks.
• Over-detailed or redundant he extraction of statements.
• Incorrectly categorizing evidence as contradictory.

Future Directions

81

Future Directions

82

Future Directions

Focusing on evaluation of complex and not well-defined tasks:

• Open-ended
• With dynamic test set
• Non-definitive ground truth

83

Future Directions

• Finding out what works for people?
• Should we set human performance as our ground truth, or aim to surpass it?
• To what extent can generative models replace human efforts in evaluation?
• What is the optimal role of humans in the evaluation process during the generative era?

​

84

85

Thanks!

Any questions?

Negar Arabzadeh

: Narabzad@uwaterloo.ca : https://www.negara.me/

: @NegarEmpr : Narabzad

​

86

Assessing Responses with Relevance Judgments

• Experiment:
• measuring the average similarity between each generated answers and the judged relevant passages from different levels of relevance.
• Findings:
• Average similarity between generated answers and passages decreases with relevance level.
• gpt-4 appears to be a more “convincing liar” compared to gpt-3.5-turbo, since it consistently yields lower similarity scores to the relevant judged passages.

​

87

Fairness

88

Fairness

• A fair ranker provides a balanced representation of the protected attributes.
• For example: gender, race, ethnicity, and age.

​

​

​

​

​

​

​

​

89

Fairness

• A fair ranker provides a balanced representation of the protected attributes.
• For example: gender, race, ethnicity, and age.

​

​

• Ideal: Given a gender-neutral query:
• The retrieved documents are Relevant to the query
• The retrieved documents are do not show inclination towards a specific gender
• Between two equally related documents, the one with lower degree of gender bias is ranked higher

​

​

​

​

​

​

​

90

Fairness

91

Fairness

92

…

…

• Gender Bias Metric: Average Rank Bias (ARaB)

ARaB: Bias Inclination toward

ARaB: Bias Inclination toward

​

​

​

93

Do Neural Ranking Models Intensify Gender Bias?

​

​

​

​

Depth of ranking

• Gender Bias Metric: Average Rank Bias (ARaB)

ARaB: Bias Inclination toward

ARaB: Bias Inclination toward

• Only the red bar is unsupervised approach (BM25). The rest are supervised neural rankers.
• All the rankers has male-inclination biases
• Neural-based rankers intensify gender biases toward male inclination more than the unsupervised ranker.

​

​

​

94

Do Neural Ranking Models Intensify Gender Bias?

​

​

​

​

Depth of ranking

Evaluation in terms of Fairness

95

Investigating Gender Biases in IR

Evaluation in terms of Fairness

96

Investigating Gender Biases in IR

Accuracy

97

Self-Evaluation through QPP

• Query Performance Prediction (QPP): Predicting the quality of retrieved documents, in satisfying the information needs behind the query.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

98

Relevant documents

Information Need

Retrieval System

Query

Self-Evaluation through QPP

• Query Performance Prediction (QPP): Predicting the quality of retrieved documents, in satisfying the information needs behind the query.

​

​

​

​

​

​

• How good is the predicted quality?

​

​

​

​

​

​

​

​

99

Relevant documents

Information Need

Retrieval System

Query

Self-Evaluation through QPP

​

​

• Applications:

​

​

​

​

100

• Query Performance Prediction (QPP) has shown to be highly correlated with retrieval performance

Self-Evaluation through QPP

​

​

• Applications:
• Proposed Methods:
• Neural embedding-based QPP
• QPP based on perturbations in query representation
• QPP based on document coherency
• Contextualized transformer-based QPP

​

​

​

​

101

• Query Performance Prediction (QPP) has shown to be highly correlated with retrieval performance

Self-Evaluation through QPP

​

​

• Applications:
• Proposed Methods:
• Neural embedding-based QPP
• QPP based on perturbations in query representation
• QPP based on document coherency
• Contextualized transformer-based QPP

​

​

​

​

102

• Query Performance Prediction (QPP) has shown to be highly correlated with retrieval performance