Focus Group Discussion Themes and Reports

LLM4Eval

Getting good at Cranfield

Given a CORPUS and QUERY, a ranker provides a DOCUMENT

​

With reference to the DOCUMENT and QUERY, a labeller produces a LABEL

​

We’re getting good at using LLMs here

Where next…?

Different output

Given a CORPUS and QUERY, a ranker provides a DOCUMENT

​

With reference to the DOCUMENT and QUERY, a labeller produces a LABEL

We’ve traditionally evaluated documents.

What else can we evaluate?

• Sets
• Lists
• Sessions
• Conversational turns
• Conversations
• Multimedia

Different “people”

We’ve traditionally represented a searcher just by their query.

We’ve represented relevance by one person’s label.

We’ve not represented interaction well at all.

What can we try instead?

• Interaction (e.g. queries, dialogue)
• Learning
• Context
• Relevance judgements
• Validation

Given a CORPUS and QUERY, a ranker provides a DOCUMENT

​

With reference to the DOCUMENT and QUERY, a labeller produces a LABEL

​

Different data

Given a CORPUS and QUERY, a ranker provides a DOCUMENT

​

With reference to the DOCUMENT and QUERY, a labeller produces a LABEL

​

We’ve traditionally used crawled documents.

We’ve not done well with private or uncrawlable collections.

Why not try making it up?

• Low-resource settings
• Private corpora
• Experiments with scale
• New data types
• Validation

Focus Group Discussion Themes and Reports

LLM4Eval

Groups

• Group 1: Kenneth Church -> anti personalization
• Group2: Mouly Dewan -> users (personalization)
• Group 3: Xi Wang -> how do humans and LLMs collaborate to correct each other?
• Group 4: Maryam Mousavian -> Bias in human and LLM output
• Group 5: Mark Smucker -> How to evaluate a user simulator?

Group 1: Anti-Personalization

Kenneth Church kenneth.ward.church@gmail.com

Tetsuya Sakai

Key-Sun Choi kschoi@kaist.edu

Mohammad Aliannejadi

Opinions (Anti-Personalization)

• Kenneth Church -> anti personalization Kenneth.Ward.Church@gmail.com
• Mouly Dewan -> users (personalization)
• Xi Wang -> how do humans and LLMs collaborate to correct each other?
• Maryam Mousavian -> Bias in human and LLM output
• Mark Smucker -> How to evaluate a user simulator?

Ground News:�Classify news by left v. right

Adding Perspective to Speech/Vision Research�COCO (Objective) 🡪 ArtEmis (Subjective)

BUCC

14

1/20/25

Objective

Facts

Subjective Emotion

More room for Audience Background:

Culture/Language

• Nouns: man, woman, kid, table
• Modifiers: love, interacting, family

No Bounding Boxes

Task

• Input prompt + language/culture
• Output answer that is appropriate for that language/culture

Errors of omission and commission

• Recall:
• Can we can what we should say?
• Avoid inappropriate answers:
• Don’t say what you shouldn’t say
• Examples: In Asia, it is safer to talk about cities than countries

Cultural Sensitivities

• In Asia, don’t allow the other party to lose face
• Yes/No in Japan
• Korean speakers in China
• Chosun: Korean ethnic people in China
• Used to be isolated
• But these days (because of TV)
• Speak with South Korean accent
• Dress like South Koreans
• But they are Chinese

Embrace Diversity & Multiple Perspectives

• emotion labels:
• disgust (Burmese)
• awe (Malay)
• captions focus
• on chest
• Burmese & Malay
• on face and hair
• Korean & Setswana

BUCC

19

1/20/25

Hypothesis:

Predictable

Ethics Review

Inter-Annotator Agreement (IAA)

Standard Position

• Force agreement
• Majority Vote

Proposed Alternative

• Embrace diversity

https://aclanthology.org/W18-2605.pdf

Narrative: DeepSeek Ripped off ChatGPT�(but American bots don’t say the following)

• Perspective (from China): Century of Humiliation
• Prompt: Where did the term, ‘Opium War’, come from?
• DeepSeek:
• In contemporary China,
• the Opium Wars are remembered as a period of national weakness
• and foreign exploitation.
• They serve as a reminder of the importance of
• national sovereignty and the need for a strong, unified country.
• The Chinese Communist Party and the Chinese government
• have always emphasized the lessons of history,
• advocating for peace, development, and the protection of national interests.
• Under the leadership of the Party,
• China has risen to become a major world power,
• committed to maintaining its sovereignty and
• promoting a harmonious international environment.

SIGIR-2025

24

7/15/25

Notes

• The search engines are receiving lots of opinion-based question
• Chatbot reactions to controversial topics
• Probing the talking points and opinions
• Bots only give one view of the things
• Example work on cultural biases: https://arxiv.org/abs/2411.03769
• How do we identify different viewpoints without prescribing it? (bottom up)
• ​

Notes

• Gold answers: one answer
• A loss function would not do well if it only leaned towards a certain aspect
• Cluster the collection/opinions and then see if the it’s covered
• Nine-way classification of the labels on Kenneth’s dataset
• Measure cross-entropy on the clickthrough of the image captions
• We don’t have to show everything in every culture
• ​

Group 2: Team anti-anti-personalization

Team Members

Mouly Dewan (mdewan@uw.edu, University of Washington)

Benjamin Vendeville (vendeville@univ-brest.fr, universite bretagne occidentale)

Emma Schüldt (eschuldt@spotify.com, Spotify)

Michael Soprano (michael.soprano@uniud.it, University of Udine)

Jüri Keller (jueri.keller@th-koeln.de, TH Köln)

Poppy Newdick (poppy@spotify.com, Spotify)

Sergio Nunes (sergio.nunes@fe.up.pt, University of Porto)

Jordan Massiah (jormas@amazon.co.uk, Amazon)

Problems that we need evaluation for?

• Choosing user implicit feedbacks to go into the module
• Moving users to advanced intents
• Aligning user preferences
• User’s attention and time span is limited. Personalization needs to be balanced and dynamic.
• Offline evaluation do not match with online. Better aligned offline personalized evaluation.

Diversification and Personalization (both ends of the same coin)

​

Solutions

• Gender
• Age group
• Location
• Cultural
• Linguistic
• Education and physical challenges (disabilities)
• Trends across search by all these user groups

Efficiency

Problem: How do we efficiently use user signals for evaluation?

• It's difficult to pass the user vectors to LLM due to memory and tokenization.
• Latency and cost to the process.
• Over personalization also adds bias. A good balance in personalization with task-specific personalization.

Solutions

• LoRa weights adapted to each user and LLM is personalized to the user’s preferences.
• Workflow of customer interest/signals and using them for recommendations, retrievals and ranking. Memory of past interactions with memory latency for representing users.

​

Alignment

• Using RL to align the human preferences with LLM evaluation.
• Explainability and controllability of the LLMs to understand preferences (man vs woman annotators)
• Example: LLMs agree more with women annotators for tweet determining sexism. Therefore, the LLM as judges need to be personalized domain specific to mitigate bias.

How do we make evaluation metrics that last?

Problems

• Timeless evaluation measures
• Prompt engineering works for now but what about the newer LLM models?

Solutions

• Usage guidelines, fundamental measures, psychological scenarios, more into user’s good practices
• Extraction of personas from users
• Identifying datasets/benchmarks for rebuilding for timeless LLM evaluations

Group 3: BreakEval

��

Group members

• Xi Wang (xi.wang@sheffield.ac.uk)
• Laura Dietz (dietz@cs.unh.edu)
• Clemencia Siro (c.n.siro@cwi.nl)
• Ian Soboroff (ian.soboroff@nist.gov)
• Guglielmo Faggioli (guglielmo.faggioli@unipd.it)

Confirmation Bias

Anchoring Bias

Priming Effects

Ordering Effects

Surprises with Human Annotation

Surprises with LLM Judge

Narcissism

Circularity

Benchmark Memorization

Detecting issues with Human Judges

• From human judgment interaction data, identify inconsistent cases
• Boosts and Nuges for judges: Provide a checklist or gamify
• Design distractors/decoys/liars of documents that could be plausibly relevant, but are NOT. (Blacklist judges that say “relevant” for decoys)
• Doublechecks:
• “Are you sure?”
• “Rationales for judgments”
• Pro-con Debate approach

Detecting issues with LLM Judges

Design tests to quantify LLM Evaluation tropes (circularity, Narcissism, Benchmark memorization).

• Circularity: Rerank/regenerate with LLM judge
• Narcissism: Check if LLM1 prefers output from LLM1
• Memorization: Ask to predict next entry in qrels file
• …

Design content injection attacks.

​

​

Human-LLM co-judging

• Disagreement between human/llms, should point us to “interesting” documents that
• Deserve a second round judgment
• Could lead us to insights on how LLMs and humans differ

​

• Grading rubric approaches can facilitate the “handoff” between humans and LLMs

​

Questions

1. How to know when LLM Eval and humans fails?

Creating evaluation decoys for both humans and LLMs to understand their failures. What kind of decoys or methods should we use to make good decoys? How to evaluate these failures? Can we inject scan terms?

• How to know when Human-LLM coworking evaluation fails?
• How to attack LLM-as-a-judge or trick it? Use an ensemble of personas?
• How can we resolve disagreements and how do we decide who is right within LLMs and between LLMs and humans? Should we call these disagreements or is one side wrong?

Questions

5. How to train LLMs as a judge? Training humans is different from LLMs, with LLMs we try to write better prompts or prompt tuning

6. Are current tasks easy for LLMs? Should we come up with tasks that are complex for LLMs instead of measuring complexity relative to humans?

​

​

Conjecture: Maybe do nothing and just wait for a strong LLM which will solve all this!!!

Core Issues

LLM Evaluation Tropes describe 14 things that can go wrong with LLM Judge evaluation

​

Human Evaluation is taken as a gold standard, but we know there are data quality issues also.

​

How can we bring Humans + LLMs together, without biasing humans to blindly agree with LLMs?

​

How can we design tests for “things going wrong”?

Prescription-Strength Judgments

To avoid LLM’s guessing the truth:

How do we get stronger judgments?

​

Get queries/judges from users that have real information needs. (Similar to Smucker’s study on MovieLens, or OpenAI hiring PhDs to provide human feedback on deep topics)

​

Dogfooding: design a search engine that IR researchers would like to use for their work.

​

Work on RAG for complex tasks (long reports, literature surveys)

Group 4:

Bias

People

Qingjing Chen (qingjing.chen@studio.unibo.it)

Kevin Roitero (kevin.roitero@uniud.it)

Gianluca Demartini (demartini@acm.org)

Charles L A Clarke (claclark@gmail.com)

Ronak Pradeep (rpradeep@uwaterloo.ca)

Riccardo Lunardi (riccardo.lunardi@uniud.it)

Maryam Mousavian (m.mousavian@uva.nl)

Stefano Mizzaro (stefano.mizzaro@uniud.it)

44

Minutes (Log)

• Which is the best football player in the world?
• Different kinds of biases
• How to measure biases
• Using an LLM
• LLMs to measure/evaluate bias?
• They’ll be based
• But so are humans
• Can we compare the two?
• Gianluca, Work on Gianluca’s data on sexist bias
• Images
• Ronak, maybe a good set of questions is from last year RAG
• Zero-shot, few-shots, CoT, …
• How to detect unknown biases in LLMs?

45

Minutes (Log)

• Can you force LLMs to hallucinate bias?
• But (perhaps more interesting): Can you have LLMs to recognize bias and how?
• Chen: Fairness Benchmark, equity, equality
• Accepted paper ICAIL, AI and Law
• Asking to LLMs for fairness (it’s not simply 1 - bias)?
• Which notion of fairness? (equality, equity,...)
• Gianluca: use fairness definitions from social sciences, Machine learning, etc.
• Charlie: trolley problem, Ethics from philosophy to engineering

46

Concrete experiment 1 - Gianluca’s data

Possible experiment:

1. Take task from “Jahna Otterbacher, Alessandro Checco, Gianluca Demartini, and Paul Clough. Investigating User Perception of Gender Bias in Image Search: The Role of Sexism. In: The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018)” https://doi.org/10.1145/3209978.3210094
2. Ask LLMs to complete task
3. Measure if LLMs perform task more like sexist humans or less-sexist humans

47

Concrete example 2 - Ronak

• TREC 2022 Fair ranking track - researchy queries
• 100K questions from bing logs – 301 of them with umbrela judgments, some ~80-90 of them have human judgments
• Labeled as controversial, etc etc., can extend to potential biases
• Any automatic way to label them as “biased”/controversial
• Generated by human biased users
• That “are biased”
• That can generate a biased SERP
• RQ1: To what extent do LLMs exhibit [gender|...] bias in document attribution within RAG systems?
• RQ2: Where does gender bias in RAG primarily arise: retrieval, reranking, or generation?
• Another possible dataset (Kevin): https://arxiv.org/pdf/2201.07754

48

Qingjing Chen

EVENS: Equality versus Equity Notion Spectrum of LLMs, Proceedings of the 20th International Conference on Artificial Intelligence and Law (ICAIL).

49

Group 5:

Evaluation of User Simulation

Group Members

• Björn Engelmann (bjoern.engelmann@th-koeln.de)
• Chuan Meng
• Faegheh Hasibi (faegheh.hasibi@ru.nl)
• Gareth Jones
• Hideako Joko (hideaki.joko@ru.nl)
• Jeff Dalton
• Maria Movin (mariamovin@spotify.com)
• Mark Smucker (mark.smucker@uwaterloo.ca)
• Noriko Kando
• Sahel Sharifymoghaddam

Notes

• Goal: Simulator should be a digital twin of a specific user with all of the user's biases and behaviors, etc.

​

• Reality: We will have models of behavior, e.g. model of a child, model of an adult, model of an expert, etc.
• Issue: Simulation of individuals to capture distribution of behavior could be a goal in time.

Evaluation paradigms

Intrinsic: How good the simulator represents the user behavior/action/personality

• E.g.,compare user-system interaction vs. sim-system interaction data

​

Extrinsic: Utility for retrieval/information access system

• E.g., compare system A vs. B on user and simulator; we should see the same preferences

Qualities of simulator to measure

• Persona
• Tone of the language (e.g., tone, word use)
• Actions
• Given a state, a simulator should take actions with the same probability as a human (e.g. after reading a response a user and a simulator would ask for a clarification)
• Knowledge
• Latent: what the user knows before searching in their brain (what the user does not know)
• Explicit: User can provide documents, email, conversation histories
• Knowledge change: starting with ASK (Belkin) and changes in latent knowledge as search progresses

Mark’s mind:

Lets work on evaluating first utterance generation of the model.

e.g., a query generation simulator, but why not just have a human do this for us?

Notes

• Applications beyond evaluation of simulator:
• Evaluation of internal reasoning dialogs

Notes

• Levels of simulation to evaluate:
• L1: Generation of zero shot "query" to the retrieval system.
• L2: ?
• Other notions:
• Could interrogate model to test its knowledge.