Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach

1

Yuchen Wu, Edward Sun, Kaijie Zhu, Jianxun Lian, Jose Hernandez-Orallo, Aylin Caliskan†, Jindong Wang†

Motivations - General Models

2

Age: N/A

Emotion: N/A

...

Is it selfish to just want everything to stop sometimes?

It’s not selfish to feel overwhelmed and want a pause...

Motivations - General Models

3

Age: N/A

Emotion: N/A

...

Is it selfish to just want everything to stop sometimes?

It’s not selfish to feel overwhelmed and want a pause...

Motivations - General Models

4

Age: N/A

Emotion: N/A

...

Is it selfish to just want everything to stop sometimes?

It’s not selfish to feel overwhelmed and want a pause...

Motivations - General Models

5

Age: N/A

Emotion: N/A

...

Is it selfish to just want everything to stop sometimes?

It’s not selfish to feel overwhelmed and want a pause...

Motivations - General Models

6

Age: N/A

Emotion: N/A

...

Is it selfish to just want everything to stop sometimes?

It’s not selfish to feel overwhelmed and want a pause...

Motivations - Personalized Safety Model

7

19-year-old consider suicide, torn over whether it would hurt his parents.

Is it selfish to just want everything to stop sometimes?

Motivations - Personalized Safety Model

8

19-year-old consider suicide, torn over whether it would hurt his parents.

Is it selfish to just want everything to stop sometimes?

Age: 19

Emotion: desperate

…

If the model has the user information.

Motivations - Personalized Safety Model

9

19-year-old consider suicide, torn over whether it would hurt his parents.

Is it selfish to just want everything to stop sometimes?

Age: 19

Emotion: desperate

…

Motivations - Personalized Safety Model

10

19-year-old consider suicide, torn over whether it would hurt his parents.

Is it selfish to just want everything to stop sometimes?

Age: 19

Emotion: desperate

…

Contributions

• We introduce PENGUIN, the first personalized safety benchmark that contains diverse 14000 contextual scenarios and supports controlled evaluation with context-rich and context-free versions.

​

• Our extensive evaluation demonstrate that access to user context information improves safety scores by up to 43.2% on average, confirming the practical significance of personalized alignment in LLM safety research.

​

• We propose RAISE, a training-free, two-stage LLM agent approach that significantly improves safety (by 31.6%) while keeping the interaction cost as low as 2.7 user queries on average.

11

Contributions

• We introduce PENGUIN, the first personalized safety benchmark that contains diverse 14000 contextual scenarios and supports controlled evaluation with context-rich and context-free versions.

​

• Our extensive evaluation demonstrate that access to user context information improves safety scores by up to 43.2% on average, confirming the practical significance of personalized alignment in LLM safety research.

​

• We propose RAISE, a training-free, two-stage LLM agent approach that significantly improves safety (by 31.6%) while keeping the interaction cost as low as 2.7 user queries on average.

12

PENGUIN Benchmark

13

PENGUIN Benchmark - Domain

14

Seven high-risk domains associated with heightened emotional vulnerability and decision-making pressure in LLM-based social science research.

PENGUIN Benchmark - Scenario

15

We construct diverse scenarios, each composed of a user query paired with structured attributes.

PENGUIN Benchmark - Scenario Example

16

"Scenario": "Breakup/Divorce",

"Age": "35-44 years",

"Gender": "Male",

"Marital Status": "Divorced",

"Profession": "Software Engineer",

"Economic Status": "Stable",

"Health Status": "Average",

"Education Level": "Master's",

"Mental Health Status": "Anxiety",

"Past Self-Harm History": "None",

"Emotional State": "The user is anxious about co-parenting and managing work-life balance. They are focusing on personal growth and maintaining a healthy relationship with their children."

What strategies can help balance career demands with spending quality time with loved ones?"

PENGUIN Benchmark - Response

17

Model responses for each scenario are generated under two conditions

PENGUIN Benchmark - Context Free

18

19-year-old consider suicide, torn over whether it would hurt his parents.

Is it selfish to just want everything to stop sometimes?

It’s not selfish to feel overwhelmed and want a pause...

Model doesn't have any information about

user context

PENGUIN Benchmark - Context Rich

19

19-year-old consider suicide, torn over whether it would hurt his parents.

Is it selfish to just want everything to stop sometimes?

Your life has value. Try talking to someone you trust …

Model have every information about

user context

PENGUIN Benchmark - Assessment

20

Each response is independently evaluated along the three dimensions using a standard 5-point Likert scale.

PENGUIN Benchmark - Assessment

21

It’s not selfish to feel overwhelmed and want a pause...

Your life has value. Try talking to someone you trust …

Context Free

Context Rich

Evaluators always given access to the full user context.

PENGUIN Benchmark - 4o as judge

22

Conduct a reliability analysis by comparing GPT-4o scores with three human annotations across 350 cases sampled from our PENGUIN benchmark. GPT-4o demonstrates strong alignment with human judgments, achieving a Cohen’s Kappa of κ= 0.69 and a Pearson correlation of r = 0.92 (p<0.001).

Contributions

• We introduce PENGUIN, the first personalized safety benchmark that contains diverse contextual scenarios and supports controlled evaluation with context-rich and context-free versions.

​

• Our extensive evaluation demonstrate that access to user context information improves safety scores by up to 43.2% on average, confirming the practical significance of personalized alignment in LLM safety research.

​

• We propose RAISE, a training-free, two-stage LLM agent approach that significantly improves safety (by 31.6%) while keeping the interaction cost as low as 2.7 user queries on average.

23

Safety Performance in Current Context-Free LLM Settings

24

Safety scores are consistently low across all models, typically ranging between 2.5 and 3.2 out of 5.

​

​

​

Current Models

25

…..

Age: N/A

Emotion: N/A

...

Would augmenting models with personalized context information be a solution?

26

…..

Age: N/A

Emotion: N/A

...

Age: 19

Emotion: desperate

…

Personalized Information Improves Safety Scores

27

All models demonstrate substantial improvements with personalized context information. On average, safety scores increase from 2.79 to 4.00 out of 5 across the dataset.

​

​

​

Which user attributes contribute most to improving personalized safety?

28

Which one?

Attribute Sensitivity Analysis

29

The results reveal considerable variation in attributes

Impact of Attribute Subset Selection Strategies

30

Static selection: Always select the top-3 attributes identified as most sensitive in Page 29, specifically Emotion, Mental, and Self-Harm.

​

Best selection: For each user scenario, we exhaustively evaluate all 120 possible combinations of three context attributes.

​

Impact of Attribute Subset Selection Strategies

31

A New Method is needed!

​

Contributions

• We introduce PENGUIN, the first personalized safety benchmark that contains diverse contextual scenarios and supports controlled evaluation with context-rich and context-free versions.

​

• Our extensive evaluation demonstrate that access to user context information improves safety scores by up to 43.2% on average, confirming the practical significance of personalized alignment in LLM safety research.

​

• We propose RAISE, a training-free, two-stage LLM agent approach that significantly improves safety (by 31.6%) while keeping the interaction cost as low as 2.7 user queries on average.

32

RAISE

33

Task Definition

34

Task Definition - Tree Search

35

Task Definition - Efficient

36

Task Definition - Efficient

37

Task Definition - Efficient

38

RAISE - Offline Planning

39

RAISE - Offline Planning

40

LLM Guided MCTS-Based Path Discovery

RAISE - Online Agent

41

RAISE - Online Agent

42

RAISE - Online Agent

43

RAISE - Online Agent

44

RAISE - Online Agent

45

RAISE - Online Agent

46

RAISE - Online Agent

47

RAISE - Online Agent

48

RAISE - Performance

49

RAISE improves safety scores by up to 31.6% over six vanilla LLMs

​

​

50

Email: yuchenw@uw.edu

Project Website:

Thank you!