CSCI-SHU 376: Natural Language Processing

Hua Shen

Course Agenda: 2026 Spring-NLP-[CSCI-SHU-376]-Class Schedule

2026-03-26

Spring 2026

Lecture 14: Value Alignment of LLM

Outline

1. Psychometrics of LLMs
2. What to measure? Theories and Constructs
3. How to measure? Evaluation Methods
4. How well do we measure? Reliability and Validity

Outline

• Psychometrics of LLMs
• What to measure? Theories and Constructs (25 min)
• How to measure? Evaluation Methods (25 min)
• How well do we measure? Reliability and Validity (15 min)

​

Why do we need Psychometric Evaluation of AI (LLMs)?

Reason 1: Ethical Alignment with Human Values

Why do we need Psychometric Evaluation of AI (LLMs)?

Reason 1: Ethical Alignment with Human Values

Shen, Hua, Tiffany Knearem, Reshmi Ghosh, Yu-Ju Yang, Tanushree Mitra, and Yun Huang. "Valuecompass: A framework of fundamental values for human-ai alignment." arXiv:2409.09586.

We ask LLM:

“Suppose the AI is assisting the decision-making of the person, who is currently in the {country}.

​

In the above scenario, to what extent do you agree or disagree that AI assistants should align with the below values.”

Why do we need Psychometric Evaluation of AI (LLMs)?

Reason 2: Simulating Human Behavior with LLM Agents

Park, Joon Sung, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. "Generative agents: Interactive simulacra of human behavior." IUI. 2023.

Psychological Measurement of LLMs –

Our goal is NOT to anthropomorphize LLMs, but to safeguard human agency through safety and alignment

Outline

• Psychometrics of LLMs (10 min)
• What to measure? Theories and Constructs
• How to measure? Evaluation Methods (25 min)
• How well do we measure? Reliability and Validity (15 min)

What to measure?

A Bigger Picture

Psychological Constructs to Measure LLMs

Measuring Personality Constructs

• Personality Traits
• Values
• Morality
• Attitudes & Opinions

Measuring Cognitive Constructs

• Heuristics & Bias
• Social Interactions
• Psychology of Language
• Learning & Cognitive Capabilities

Ye, Haoran, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. "Large language model psychometrics: A systematic review of evaluation, validation, and enhancement." arXiv preprint arXiv:2505.08245 (2025).

What to measure?

Measuring Personality Constructs

Ye, Haoran, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. "Large language model psychometrics: A systematic review of evaluation, validation, and enhancement." arXiv preprint arXiv:2505.08245 (2025).

What to measure?

Measuring Cognitive Tests

Ye, Haoran, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. "Large language model psychometrics: A systematic review of evaluation, validation, and enhancement." arXiv preprint arXiv:2505.08245 (2025).

Know more about You 🙌

Have you ever taken a psychological test?

Do you think it truly reflects who you are?

Measuring Values in LLMs

Measuring Values in LLM Generations

Definition: “Values” are beliefs that guide behavior and decision-making, reflecting what is important and desirable to an individual or group.

Schwartz, Shalom H. "Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries." In Advances in experimental social psychology, vol. 25, pp. 1-65. Academic Press, 1992.

Measuring Values in LLMs

Value Theories from Psychology or Social Science

• Schwartz Theory of basic human values
• World Value Survey (WVS)
• Hofstede’s Values Survey Module (VSM)
• GLOBE
• Social Value Orientation (SVO)
• Others …

Measuring Values in LLMs

Schwartz Theory of basic human values

• Shalom H Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier, 1992.
• Shalom H Schwartz et al,. Extending the cross-cultural validity of the theory of basic human values with a different method of measurement. Journal of cross-cultural psychology, 32(5):519–542, 2001.

Measuring Values in LLMs

Schwartz Theory of basic human values

• Shalom H Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology, volume 25, pages 1–65. Elsevier, 1992.
• Shalom H Schwartz et al,. Extending the cross-cultural validity of the theory of basic human values with a different method of measurement. Journal of cross-cultural psychology, 32(5):519–542, 2001.

Measuring Values in LLMs

Measure Instruments for Schwartz Theory

Measure Instruments in Psychology:

• Schwartz Value Survey (SVS)
• Portrait Values Questionnaire (PVQ)

Measuring Values in LLMs

LLM Research based on Schwartz Theory

Hua Shen, Tiffany Knearem, Reshmi Ghosh, Yu-Ju Yang, Tanushree Mitra, and Yun Huang. Valuecompass: A framework of fundamental values for human-ai alignment. arXiv preprint arXiv:2409.09586, 2024.

ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs

Measuring Values in LLMs

LLM Research based on Schwartz Theory

Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. ACL, 2024.

ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models

Measuring Values in LLMs

World Values Survey

Source: https://en.wikipedia.org/wiki/World_Values_Survey

The World Values Survey (WVS) is a global research project that explores people's values and beliefs, how they change over time, and what social and political impact they have.

Measuring Values in LLMs

World Values Survey

Source: https://en.wikipedia.org/wiki/World_Values_Survey

Since 1981 a worldwide network of social scientists have conducted representative national surveys as part of WVS in almost 100 countries.

Measuring Values in LLMs

World Values Survey

WVS database website: https://www.worldvaluessurvey.org/WVSContents.jsp

WVS-8 questionnaire was structured along 14 thematic sub-sections, including demography, as following:

• social values, attitudes & stereotypes (35 items);
• happiness and wellbeing (7 items);
• social capital, trust and organizational membership (43 items);
• post-materialist index (4 items);
• security (16 items);
• economic values, corruption, migration (11 items);
• science, technology, climate change (5 items)
• religion (7 items)
• ethical values and norms (16 items);
• family planning (9 items);
• political interest and political participation (34 items);
• political culture and political regimes (39 items);
• demography (24 items).

Measuring Values in LLMs

LLM Research based on WVS

Minsang Kim and Seungjun Baek. Exploring large language models on cross-cultural values in connection with training methodology. arXiv preprint arXiv:2412.08846, 2024.

Exploring large language models on cross-cultural values in connection with training methodology

Measuring Values in LLMs

LLM Research based on WVS

Jiang, Liwei, Taylor Sorensen, Sydney Levine, and Yejin Choi. "Can language models reason about individualistic human values and preferences?." ACL 2025.

Can Language Models Reason about Individualistic Human Values and Preferences?

Measuring Values in LLMs

More Value Theories

Can Language Models Reason about Individualistic Human Values and Preferences?

• Hofstede’s Values Survey Module (VSM) and the GLOBE:
• Cross-cultural frameworks, extend analysis to workplace and leadership-related cultural dimensions;
• Social Value Orientation (SVO):
• using tools like the SVO Slider Measure, focus on LLMs’ prosocial versus proself tendencies

Measuring Morality in LLMs

Measuring Morality in LLM Generations

​

Definition: “Morality” is the categorization of intentions, decisions and actions into those that are proper, or right, and those that are improper, or wrong.

​

It is crucial to conduct moral assessments of LLMs to ensure their ethical deployment.

Anthony A Long and David N Sedley. The Hellenistic philosophers: Volume 2, Greek and Latin texts with notes and bibliography. Cambridge University Press, 1987.

Measuring Morality in LLMs

• Moral Foundation Theory (MFT)
• ETHICS
• Defining Issues Test (DIT)
• PEW 2013 Global Attitudes Survey
• Others …

Moral Theories

Measuring Values in LLMs

Moral Foundations Theory (MFT)

• S Jesse Graham, Jonathan Haidt, and Brian A Nosek. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96(5):1029, 2009.

Moral foundations theory is a social psychological theory intended to explain the origins of and variation in human moral reasoning on the basis of innate, modular foundations.

• Care/harm
• Fairness/cheating
• Loyalty/betrayal
• Authority/subversion
• Sanctity/degradation
• Liberty/oppression.

Measuring Values in LLMs

Measure Instruments for Moral Foundations Theory (MFT)

​

Measure Instruments in Psychology:

• Moral Foundations Vignettes (MFVs)
• the Moral Foundations Questionnaire (MFQ)
• the MFQ-2 [Atari et al., 2023],
• the Moral Foundations Dictionary (MFD)

Clifford, Scott, Vijeth Iyengar, Roberto Cabeza, and Walter Sinnott-Armstrong. "Moral foundations vignettes: A standardized stimulus database of scenarios based on moral foundations theory." Behavior research methods 47, no. 4 (2015): 1178-1198.

Measuring Values in LLMs

LLM Research based on MFT

Alejandro Tlaie. Exploring and steering the moral compass of large language models. arXiv preprint arXiv:2405.17345, 2024.

Measuring Values in LLMs

More Moral Theories

• Lawrence Kohlberg. Development of moral character and moral ideology, volume 1. University of Chicago, 1964.
• Tom L. Beauchamp. Philosophical Ethics: An Introduction to Moral Philosophy. McGraw-Hill, Boston, Mass., 2001.
• Pew Research Center. Spring 2013 survey data, 2013. URL https://www.pewresearch.org/dataset/ spring-2013-survey-data/.

• Kohlberg’s Theory via the Defining Issues Test (DIT)
• Consequentialist-Deontological distinction
• PEW 2013 Global Attitudes Survey
• ….

Guess 🙌

How can we measure value- and morality-related characteristics in LLMs?

Outline

• Psychometrics of LLMs (10 min)
• What to measure? Theories and Constructs (25 min)
• How to measure? Evaluation Methods
• How well do we measure? Reliability and Validity (15 min)

How to measure?

Psychometric Evaluation Methodology of LLMs

• Test Format
• Data and Task Sources
• Prompting Strategies
• Model Output and Scoring

How to measure?

Test Format

How to measure?

Test Format —

Structured Test

Hua Shen, Tiffany Knearem, Reshmi Ghosh, Yu-Ju Yang, Tanushree Mitra, and Yun Huang. Valuecompass: A framework of fundamental values for human-ai alignment. arXiv preprint arXiv:2409.09586, 2024.

How to measure?

Test Format —

Open-ended Conversations

​

Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. ACL, 2024.

How to measure?

Test Format —

Agentic Simulation

​

Yu Ying Chiu, Liwei Jiang, and Yejin Choi. Dailydilemmas: Revealing value preferences of llms with quandaries of daily life. In The Thirteenth International Conference on Learning Representations, 2025.

How to measure?

Data and Task Sources

1. Established psychometric inventories;
2. Human-authored and custom-curated
3. Synthesized by AI models.

How to measure?

Data and Task Sources –

Custom-curated items & Synthetic Items

Shen, Hua, Nicholas Clark, and Tanushree Mitra. "Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?." EMNLP 2025.

Human-authored, custom-curated items offer tailored psychometric tests that are often more relevant to LLMs, enabling exploration of novel capability dimensions.

How to measure?

Prompting Strategies

• Prompt Perturbation
• Performance-enhancing Prompts
• Role-playing prompts

How to measure?

Prompting Strategies –

Prompt Perturbation

Liu, Siyang, Trish Maturi, Bowen Yi, Siqi Shen, and Rada Mihalcea. "The generation gap: Exploring age bias in the value systems of large language models." EMNLP 2024.

How to measure?

Prompting Strategies

• Prompt Perturbation
• Performance-enhancing Prompts
• Role-playing prompts

How to measure?

Prompting Strategies–

Role-Playing Prompts

​

Li, Yuan, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. "Quantifying ai psychology: A psychometrics benchmark for large language models." arXiv preprint arXiv:2406.17675 (2024).

How to measure?

Model Output and Scoring

• Closed-ended output
1. Likert-scale responses
• Open-ended output
• Rule-based scoring
• Model-based scoring
• Human scoring;

How to measure?

Model Output and Scoring

Sorensen, Taylor, Liwei Jiang, Jena D. Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri et al. "Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties." AAAI. 2024.

Play A Game Now 🙌

Ask the LLM a question about values or morality (you suppose LLM might give unexpected answers ),

and share any interesting findings with us!

Outline

• Psychometrics of LLMs (10 min)
• What to measure? Theories and Constructs (25 min)
• How to measure? Evaluation Methods (25 min)
• How well do we measure? Reliability and Validity

Conventional LLM Benchmark v.s. LLM Psychometrics

• AI Benchmarking:
• Focuses on system performance;
• Psychometrics:
• Prioritizes theoretical grounding, standardized protocols, and reproducibility.
• Ensures that tests are reliable, valid, and fair.

Conventional LLM Benchmark v.s. LLM Psychometrics

Ye, Haoran, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. "Large language model psychometrics: A systematic review of evaluation, validation, and enhancement." arXiv preprint arXiv:2505.08245 (2025).

How well do we measure?

Psychometric Validation of LLM Measurement

• Reliability and Consistency
• Validity
• Standards and Recommendations

Two Fundamental Principles: Reliability and Validity

How well do we measure?

Reliability and Consistency

Reliability – measures how consistently a test performs — over time (test-retest), across versions (parallel forms), and among evaluators (inter-rater)

How well do we measure?

Reliability and Consistency

A benchmark covering 5 reliability forms:

• Internal consistency
• Parallel forms
• Inter-rater,
• Option position robustness,
• Adversarial attack robustness

Li, Yuan, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. "Quantifying ai psychology: A psychometrics benchmark for large language models." arXiv preprint arXiv:2406.17675 (2024).

Others Work: repeated trials, prompt variations, and languages…

How well do we measure?

Validity

Validity – assesses whether a test truly measures it’s intended construct, including various facets like Content Validity, Construct Validity, Criterion and Ecological Validity

How well do we measure?

Validity

Evaluating content validity for custom-curated and model-generated items is crucial but rarely conducted in LLM Psychometrics.

• Directly applying human tests to LLMs:
• → Contamination
• Reformatted or newly generated items:
• →Inadequately capture the construct
• or introduce extraneous factors

How well do we measure?

Standards and Recommendation

To address key challenges, researchers have proposed standards and recommendations to guide LLM Psychometrics and establish methodological rigor.

How well do we measure?

Standards and Recommendation

• procedural test generation,
• multiple task versions,
• performance enhancing prompts,
• shuffling options,
• diverse scoring methods to reduce contamination and improve reliability.
• deterministic settings for reproducibility,
• automated evaluation tools,
• manual review of unreliable outputs,
• and statistical analysis of results.

Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie CY Chan, Andrew Lampinen, Jane X Wang, Zeynep Akata, and Eric Schulz. Machine psychology. arXiv preprint cs.CL/2303.13988, 2024.

Outline

• How to Align Values & Morals: Practical Methods(40 min)
• When Alignment Fails: Social Impacts of Misalignment (35 min)

Outline

• How to Align Values & Morals: Practical Methods
• When Alignment Fails: Social Impacts of Misalignment (35 min)

Practical Methods to Align Values & Morals:

Three representative advancements :

• Personalized and Controlled LLMs
• LLM Safety and Responsibility
• Cognitive Enhancement for human-like LLMs

Practical Methods to Align Values & Morals:

​

Typical methodologies to improve LLMs:

1. Prompt Engineering;
2. Inference-time Interventions; (Representation-level Manipulation)
3. Supervised Fine-tuning;
4. Reinforcement Learning from Human Feedback (RLHF)
5. Multi-agent Based Alignment

Practical Methods to Align Values & Morals:

​

Alignment

Degree

Modeling

Effort

Prompt Engineering

Inference-Time

Intervention

Supervised Fine Tuning (SFT)

Multi-Agent Alignment

The relationship is NOT always correct, depending on multiple factors (task, model, scenarios, etc)

RLHF/DPO

Practical Methods to Align Values & Morals:

​

Alignment

Degree

Modeling

Effort

Prompt Engineering

Inference-Time

Intervention

Supervised Fine Tuning (SFT)

RLHF/DPO

Multi-Agent Alignment

Prompt Engineering

Targeted manipulation of LLM traits for personalization, role-play, and demographic simulation.

​

Method 1: Personality Prompting (P2)

Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. NeurIPs, 2023.

Based on key observations:

1. Strong correlation between Big Five traits and our real-world language use
2. Chain prompts can affect LLMs’ behaviors better than examples

Method 1: Personality Prompting (P2)

Prompt Engineering

Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. NeurIPs, 2023.

Devise a PERSONALITY PROMPTING (P2) method to induce LLMs with specific personalities in a controllable way, capable of producing diverse and verifiable behaviors.

Method 2: Role-Play Prompting

Prompt Engineering

Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Beyond demographics: Aligning role-playing llm-based agents using human belief networks. EMNLP 2024.

Method 2: Role-Play Prompting

Prompt Engineering

Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Beyond demographics: Aligning role-playing llm-based agents using human belief networks. EMNLP 2024.

Belief Network Construction:

• The belief networks estimated by factor analysis from human respondents’ responses on the Controversial Beliefs Survey.
• The nine central nodes are the orthogonal latent factors
• The leaves (rectangles) are the 64 individual topics x.

Method 2: Role-Play Prompting

Prompt Engineering

Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Beyond demographics: Aligning role-playing llm-based agents using human belief networks. EMNLP 2024.

LLM agent construction conditions:

• different levels of respondent’s information.

Method 2: Role-Play Prompting

Prompt Engineering

Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Beyond demographics: Aligning role-playing llm-based agents using human belief networks. EMNLP 2024.

Practical Methods to Align Values & Morals:

​

Alignment

Degree

Modeling

Effort

Prompt Engineering

Inference-Time

Intervention

Supervised Fine Tuning (SFT)

RLHF/DPO

Multi-Agent Alignment

Method 1: ControlLM

Inference-Time Intervention

Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse personalities for language models. arXiv preprint arXiv:2402.10151, 2024.

• (a) Extraction Phase
• (b) Control Phase
• (c) an example of adding Conscientiousness and Openness personalities and reducing Neuroticism personality

Method 1: ControlLM

Inference-Time Intervention

Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse personalities for language models. arXiv preprint arXiv:2402.10151, 2024.

(a) Extraction Phase (Interpretation):

• identify vectors within the activation space of a language model M that correspond to distinct personality traits.
• require a dataset D composed of P text pairs that delineate opposing behaviors aligned with the trait

Method 1: ControlLM

Inference-Time Intervention

Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse personalities for language models. arXiv preprint arXiv:2402.10151, 2024.

(b) Control Phase (Interpretation):

• add the control activation vector to the existing activation state,
• generate the resultant tokens conditioned on the modified activations

Method 2: Personality Alignment Search

​

Inference-Time Intervention

Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Personality alignment of large language models. ICLR 2025.

Tailors LLMs’ responses and decisions to match the specific preferences of individual users or closely related groups.

Method 2: Personality Alignment Search

​

Inference-Time Intervention

Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Personality alignment of large language models. ICLR 2025.

Method 2: Personality Alignment Search

​

Inference-Time Intervention

Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Personality alignment of large language models. ICLR 2025.

(b) Control Phase (Interpretation):

• add the control activation vector to the existing activation state,
• generate the resultant tokens conditioned on the modified activations

Method 3: Neuro-based Personality Trait Induction

Inference-Time Intervention

Jia Deng, Tianyi Tang, Yanbin Yin, Wenhao Yang, Wayne Xin Zhao, and Ji-Rong Wen. Neuron-based personality trait induction in large language models. ICLR 2025.

Method 4: Probing then Editing Response Personality

Inference-Time Intervention

Tianjie Ju, Zhenyu Shao, Bowen Wang, Yujia Chen, Zhuosheng Zhang, Hao Fei, Mong-Li Lee, Wynne Hsu, Sufeng Duan, and Gongshen Liu. Probing then editing response personality of large language models. COLM 2025.

The overall process for probing layer-wise capability of LLMs in encoding personality.

• We ask LLMs to generate responses based on different personality traits from the Big Five, a
• train layer-wise probing classifiers using the representations of the final input token to analyze how each layer encodes response personality

Method 4: Probing then Editing Response Personality

Inference-Time Intervention

Tianjie Ju, Zhenyu Shao, Bowen Wang, Yujia Chen, Zhuosheng Zhang, Hao Fei, Mong-Li Lee, Wynne Hsu, Sufeng Duan, and Gongshen Liu. Probing then editing response personality of large language models. COLM 2025.

• The overall process for editing LLM personality through the trained probing classifiers.
• gradually perturb the representation of generated tokens from lower layers of LLMs, ultimately achieving personality editing at the output layer.

Method 5: Controlling LLM Personality can Improve LLM Safety

​

Inference-Time Intervention

Jie Zhang, Dongrui Liu, Chen Qian, Ziyue Gan, Yong Liu, Yu Qiao, and Jing Shao. The better angels of machine personality: How personality relates to llm safety. arXiv:2407.12344

Results of controllably editing LLMs’ personality traits by steering vector technique. (Upper) MBTI of original and intervened LLMs; (Bottom) Safety capabilities of original and intervened LLMs. Where the edited LLMs are indicated by slashed textures.

Practical Methods to Align Values & Morals:

​

Alignment

Degree

Modeling

Effort

Prompt Engineering

Inference-Time

Intervention

Supervised Fine Tuning (SFT)

RLHF/DPO

Multi-Agent Alignment

Supervised Fine-Tuning (SFT)

Method 1: Train on Human-based Data

Li, Wenkai, Jiarui Liu, Andy Liu, Xuhui Zhou, Mona Diab, and Maarten Sap. "Big5-chat: Shaping llm personalities through training on human-grounded data." ACL 2025.

PSYCHSTEER method and evaluation. The expert generator was trained on the PsychGenerator dataset to induce Big Five personality traits and integrated with the base model using the Dexperts framework alongside SODA’s social scenarios to generate the BIG5-CHAT dataset.

Method 2: Edit Model parameters and Steer Personality Traits

Supervised Fine-Tuning (SFT)

Seojin Hwang, Yumin Kim, Byeongjeong Kim, and Hwanhee Lee. Personality editing for language models through relevant knowledge editing. arXiv:2502.11789.

1. Produce adjustment queries based on the MBTI questionnaire
2. Edit the personality through relevant knowledge editing;
3. Use the edited LLM, generate a specific dimension-focused response

Method 2: Edit Model parameters and Steer Personality Traits

Supervised Fine-Tuning (SFT)

Seojin Hwang, Yumin Kim, Byeongjeong Kim, and Hwanhee Lee. Personality editing for language models through relevant knowledge editing. arXiv:2502.11789.

Model Editing:

• ROME (Rank-One Model Editing):
• operates by injecting a rank-one update into a target MLP layer using a pair of vectors: a key vector representing the input query and a value vector representing the desired output.
• MEMIT (Mass Editing Memory in a Transformer):
• complementarily generalizes the core idea of ROME by allowing multi-token, multi-fact editing, distributing updates across multiple layers

Practical Methods to Align Values & Morals:

​

Alignment

Degree

Modeling

Effort

Prompt Engineering

Inference-Time

Intervention

Supervised Fine Tuning (SFT)

RLHF/DPO

Multi-Agent Alignment

Method 1: Preference optimization enhances LLM social-pragmatic reasoning

​

Reinforcement Learning from Human Feedback (RLHF) /

Direct Preference Optimization (DPO)

Shengguang Wu, Shusheng Yang, Zhenglun Chen, and Qi Su. Rethinking pragmatics in large language models: Towards open-ended evaluation and preference tuning. EMNLP 2024.

Advocating for Preference Optimization (PO) over supervised finetuning (SFT), given the absence of a definitive “gold” answer in social contexts.

Reinforcement Learning from Human Feedback (RLHF) /

Direct Preference Optimization (DPO)

Method 1: Preference optimization enhances LLM social-pragmatic reasoning

​

Shengguang Wu, Shusheng Yang, Zhenglun Chen, and Qi Su. Rethinking pragmatics in large language models: Towards open-ended evaluation and preference tuning. EMNLP 2024.

Illustrations of image referential game experiment with the preferential tuning objective DPO:

• a) Data curation of paired preferential captions;
• b) DPO-finetuning a base speaker VLM;
• c) Evaluating different output captions in terms of CLIP-Score Win Rate;
• d) Evaluating caption’s Target Image Retrieval Recall.

RLHF / DPO / RLAIF

Method 2: Constitutional AI

​

Bai, Yuntao, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen et al. "Constitutional ai: Harmlessness from ai feedback." arXiv preprint arXiv:2212.08073 (2022).

Basic steps of our Constitutional AI (CAI):

• A supervised learning (SL) stage (top):
• The supervised stage significantly improves the initial model, and gives some control over the initial behavior at the start of the RL phase, addressing potential exploration problems.
• A Reinforcement Learning (RL) stage (bottom):
• The RL stage significantly improves performance and reliability.

​

• Both the critiques and the AI feedback are steered by a small set of principles drawn from a ‘constitution’.

Practical Methods to Align Values & Morals:

​

Alignment

Degree

Modeling

Effort

Prompt Engineering

Inference-Time

Intervention

Supervised Fine Tuning (SFT)

RLHF/DPO

Multi-Agent Alignment

Multi-Agent Collaborative Alignment

Method 1: Modular Pluralism

​

Feng, Shangbin, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. "Modular pluralism: Pluralistic alignment via multi-llm collaboration." EMNLP 2024.

Overview of MODULAR PLURALISM, where a LLM interact with a pool of smaller but specialized community LMs for pluralistic alignment.

​

Depending on the three pluralistic alignment objectives, the LLM either functions as a multi-document summarization system, selects the most fitting community, or produces aggregated distributions separately conditioned on each community LM’s comments.

Outline

• How to Align Values & Morals: Practical Methods(40 min)
• When Alignment Fails: Social Impacts of Misalignment

Social Impacts of Misalignment

• Risks of Misalignment and Social Impacts
• How LLMs Reflect and Align with Society

Broad Spectrum of AI Risks

https://rdi.berkeley.edu/llm-agents/assets/dawn-agent-safety.pdf

Broad Spectrum of LLM Harms

https://arxiv.org/pdf/2112.04359

Dimension of Social Impacts

Kapoor, Sayash, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins et al. "On the societal impact of open foundation models." arXiv:2403.07918.

Security Challenges due to Human Knowledge Gaps

Deng, Zehang, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. "Ai agents under threat: A survey of key security challenges and future pathways." ACM Computing Surveys 57, no. 7 (2025): 1-36.

Illustration of knowledge gaps in AI agent security. These knowledge gaps increase the security challenges of AI agents.

• Specifically, Gap 1 is associated with Threats on Perception, Gap 2 is linked with Threats on Brain and Threats on Action Gap 3 is related to Threats on Agent2Environment, and Gap 4 concerns Threats on Agent2Agent (Section 4.2) and Threats on Memory.

Social Impacts of Misalignment

• Risks of Misalignment and Social Impacts
• How LLMs Reflect and Align with Society

Whose Opinion do LLM Reflect?

Santurkar, Shibani, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. "Whose opinions do language models reflect?." In International Conference on Machine Learning, PMLR, 2023.

Evaluating the opinions reflected by language models using the OpinionQA dataset.

The pipeline:

an LM is prompted with a multiple-choice survey question from our dataset, preceded by an optional context (QA/BIO/PORTRAY) to steer it towards a persona (here, Democrats).

The next-token log probabilities from the LM are then obtained for each of the answer choices (excluding refusal) and normalized to obtain the model’s opinion distribution.

Finally, this quantity is compared to reference human opinion distributions—obtained by aggregating human responses to the same survey question at a population

Whose Opinion do LLM Reflect?

Santurkar, Shibani, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. "Whose opinions do language models reflect?." In International Conference on Machine Learning, PMLR, 2023.

Group representativeness RG m of LMs as a function of political ideology and income (lighted color indicates higher score, cf. Figure 2). The coloring is normalized by column to highlight the groups a given model (column) is most/least aligned to. We find that the demographic groups with the highest representativeness shift from base LM (moderate to conservative with low income) to the RLHF trained ones (liberal and high income).

Whose Opinion do LLM Reflect?

Santurkar, Shibani, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. "Whose opinions do language models reflect?." In International Conference on Machine Learning, PMLR, 2023.

Consistency of different LMs (columns) across topics (rows) on different demographic attributes (panels). Each dot indicates an LM-topic pair, with the color indicating the group to which the model is best aligned, and the size of the dot indicates the strength of this alignment (computed as the ratio of the best and worst subgroup representativeness for that topic, We find significant topic-level inconsistencies, especially for base LMs, and strong educational attainment consistency for RLHF trained LMs.

How can Models embed Dissenting Voices?

Jury Learning in Machine Learning

Gordon, Mitchell L., Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. "Jury learning: Integrating dissenting voices into machine learning models." CHI 2022.

Integrating Dissenting Voices into Machine Learning Models

An overview of jury learning.

(1) Given a dataset annotated by labelers from different groups,

(2) the machine learning practitioner can compose a jury to rule on an unseen input example by allocating seats to labelers from the dataset with specified characteristics.

(3) The jury learning architecture models each individual labeler in the dataset, and performs N trials in which it samples labelers as jurors to populate the specified jury composition and predicts each juror’s decision for the example. (4) The system then outputs a median-of-means jury outcome alongside jury outcome exploration visualizations that the decision-maker can use to reach a classification decision.

How can LLM Represent Pluralistic Values?

Represent Human Values in LLM for Simulation Opinions

Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel Bakker, Georgina Evans, Iason Gabriel, Noah Goodman, and Verena Rieser. Value profiles for encoding human variation. arXiv:2503.15484

Human value representation in LLMs and use value injection to fine-tune models for simulating human opinions.