From Preferences

to Personas:

Personalizing Legal AI

Esmie Hurd & Ashley Jun

Why Personas?

An Under-Examined Assumption

An Example LLM Prompt:

Persona prompting is widely used, but not thoroughly validated

​

Its effectiveness is often assumed rather than demonstrated.

There’s a reason for this.

​

• No straightforward evaluation
• Most datasets are synthetic, and we don’t want cyclic validation

Use Case: Negotiations

Dataset

The CASINO Corpus

Real conversations between two people in a negotiating scenario: splitting camping supplies.

No synthetic data.

Naturally occurring behavior.

Why this dataset?

• Real dialogue
• End to end negotiations
• Multiple conversations per speaker
• Consistent IDs across conversations

Goal: Define best practices for building negotiation personas

​

Validation Method: Predict a speaker’s next move given their persona

​

Focus: Framing – how we present persona content to the LLM

Structuring Dataset for Validation

Label

Tag each

utterance with one or more moves

Taxonomy

Define a discrete set of negotiation moves

“Hey, how are you today?”�[small-talk]

“Good, can’t wait for this trip! I will need lots of extra food.”�[small-talk, self-need]

“I’ll give you all the food if you give me all the water and wood.”�[full-offer]

“Sounds like a deal!”

[full-accept]

Naive Approach: Self Reported Identities

Existing Classification Methods for Negotiators

​

• Social Value Orientation (SVO)
• Prosocial vs Proself
• Big 5 Personality Test
• Thomas-Kilmann Conflict Mode Instrument (TKI)

Example SVO-based LLM Prompt

​

“You are a Proself negotiator.

​

{Proself Definition}

{Move Options/Definitions}

{Current Negotiation Transcript}

​

Given the transcript of the negotiation so far, choose your next move(s).”

Performance of SVO Personas – Predicting Moves

F1

Naive Approach: Predicted Move Distribution

LLM acts very differently as prosocial than as proself. Huge gaps in promote-coordination, reject, and showing-empathy

Predicted Distribution of Moves by SVO Identity

Count

Naive Approach: Actual Move Distribution

But prosocial and proself negotiators don’t really act differently from one another. With SVO, LLM acts like a caricature.

Actual Distribution of Moves by SVO Identity

Count

New Grouping Approach

We want to improve framing.

But first we need to sort speakers into groups that are actually different from one another.

​

Changing framing won’t do anything if the underlying groups are essentially the same.

1. Design Transition Matrices

For each speaker, capture how they tend to respond to different moves.

2. Cluster the Matrices

Sort speakers into two groups with maximum variance.

3. Name Each Cluster

Ask LLM to give a natural language description of each group.

New Grouping Results

Framing O

Key Takeaway:

​

While self-reported SVO groups have nearly identical move distributions, cluster 1 and cluster 2 show clear behavioral differences.

Cluster Names:

​

1. “Priority Lock-in”

2. “Pattern Bundler”

​

We will use these names and descriptions as our profile content, then optimize framing.

Actual Distribution of Moves by SVO Identity

Count

Actual Distribution of Moves by Cluster

Count

New Grouping Results

Framing O

Again, our clusters show more differentiation than SVO assignment

​

• Only a 1.2% difference in acceptance rate between prosocial and proself
• Nearly 10% difference between cluster 1 and 2

​

Now, we can shift to framing. What’s the best way to present these cluster descriptions to the LLM?

Simplifying the task even more

​

Given the negotiation context leading up to an offer, can we predict whether a speaker will accept it?

Actual % of Offers Accepted

By SVO Identity

By Cluster

Framing Improvement Approach

2. Apply them to a Base Prompt

3. Test with Predictions

4. Optimize

Mutations that work well get applied again

5. Evaluate

predict on hold out data with and without the persona

1. Collect Possible “Mutations”

20+ potential improvements from several papers

Process inspired by “Promptbreeder”

(Fernando et al., 2023)

​

Treats prompts as an evolving population that

are iteratively generated, tested, and mutated

So we have several iterations of the task prompt

Base Prompt:

​

“Persona: {persona}

Negotiation so far: {transcript}

Do you accept the offer?”

See how well each prompt incorporates the persona

Performance Gains from Framing Improvement

Improved template beats default template and no-persona baseline on holdout data.

Key Takeaways

​

1. Framing Matters
1. Optimized framing gives more accurate predictions
2. Without it, baseline wins

​

1. Task-First Priming
1. The “winning” mutation involved putting the task first and the persona last

Performance of Cluster Based Personas – Predicting Acceptance

F1

Takeaways and Thoughts

Main Takeaway: Persona prompting requires more rigorous evaluation, especially in the legal field.

Framing vs. Content

Extrapolating Insights

What can we learn more broadly about personas from studying them in this specific context?

Additional Directions

1. Organizing and leveraging real, messy data

2. Pairwise preference data to extract features

This particular project focused on how we present a persona to an LLM.

But we also need to scrutinize what that persona actually contains.