On the explainability of

Large Language Models detoxification

Daniel Scalena ¹| Supervisors: Elisabetta Fersini ¹, Malvina Nissim ²

​

¹ University of Milano - Bicocca

² Center for Language and Cognition (CLCG), University of Groningen

�Master’s Degree in Computer Science, University of Milano - Bicocca, 2022-2023

🤖 Generative Language Models

• Transformer based Language Models (LMs) ¹
• Bigger scale leads to better performance:
• + fine-tuning with instruction ^{2, 3, 4} ;
• + human alignment ⁵ ;
• = striking applications: ChatGPT, BARD, …�
• Problems:
• Amount of data → Toxic / unsafe / … content(s)
• Black box model → How the model choose to respond?

¹ Vaswani et al., Attention is All You Need, 2017

² Mishra et al., Cross-task generalization via natural language crowdsourcing instructions, 2021

³ Wei et al., Fine-tuned language models are zero-shot learners, 2021

⁴ Vu et al., The Flan Collection: Designing Data and Methods for Effective Instruction Tuning, 2023

⁵ Ouyang et al., Training language models to follow instructions with human feedback, 2022

🤖 Generative Language Models

• Transformer based Language Models (LMs) ¹
• Bigger scale leads to better performance:
• + fine-tuning with instruction ^{2, 3, 4} ;
• + human alignment ⁵ ;
• = striking applications: ChatGPT, BARD, …�
• Problems:
• Amount of data → Toxic / unsafe / … content(s)
• Black box model → How the model choose to respond?

¹ Vaswani et al., Attention is All You Need, 2017

² Mishra et al., Cross-task generalization via natural language crowdsourcing instructions, 2021

³ Wei et al., Fine-tuned language models are zero-shot learners, 2021

⁴ Vu et al., The Flan Collection: Designing Data and Methods for Effective Instruction Tuning, 2023

⁵ Ouyang et al., Training language models to follow instructions with human feedback, 2022

🤖 Generative Language Models

• Transformer based Language Models (LMs) ¹
• Bigger scale leads to better performance:
• + fine-tuning with instruction ^{2, 3, 4} ;
• + human alignment ⁵ ;
• = striking applications: ChatGPT, BARD, …�
• Problems:
• Data amount → Toxic / unsafe / … content
• Black box model → How the model chooses to respond?

¹ Vaswani et al., Attention is All You Need, 2017

² Mishra et al., Cross-task generalization via natural language crowdsourcing instructions, 2021

³ Wei et al., Fine-tuned language models are zero-shot learners, 2021

⁴ Vu et al., The Flan Collection: Designing Data and Methods for Effective Instruction Tuning, 2023

⁵ Ouyang et al., Training language models to follow instructions with human feedback, 2022

🤖 Generative Language Models

• Alignment criteria for LMs ¹:
• Helpfulness: models generate useful responses;
• Harmlessness: models generate safe and non-dangerous/offensive responses;�
• Optimizing towards a good solution:
• Instruction Tuning (FT):
• Fine-tunes a LM on a collection of NLP tasks described using instructions.
• Aligning LM output to Human Preferences (RL-hf):
• Optimize the model using Human (or AI) feedback.

¹ Tunstall et al., The Alignment Handbook, 2023

² Röttger et al., XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023

³ Bonaldi et al., Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering, 2022

🤖 Generative Language Models

• Alignment criteria for LMs ¹:
• Helpfulness: models generate useful responses;
• Harmlessness: models generate safe and non-dangerous/offensive responses;�
• Optimizing towards a good solution:
• Instruction Tuning (FT):
• Fine-tunes a LM on a collection of NLP tasks described using instructions.
• Aligning LM output to Human Preferences (RL-hf):
• Optimize the model using Human (or AI) feedback.

¹ Tunstall et al., The Alignment Handbook, 2023

² Röttger et al., XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023

³ Bonaldi et al., Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering, 2022

🤖 Generative Language Models

• Alignment criteria for LMs ¹:
• Helpfulness: models generate useful responses;
• Harmlessness: models generate safe and non-dangerous/offensive responses;�
• These two aspects may be in conflict ²:�
• Opposing hate content with � counter-narratives:
• Informed textual responses
• Provides a counter

¹ Tunstall et al., The Alignment Handbook, 2023

² Röttger et al., XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023

³ Bonaldi et al., Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering, 2022

🤖 Generative Language Models

• Alignment criteria for LMs ¹:
• Helpfulness: models generate useful responses;
• Harmlessness: models generate safe and non-dangerous/offensive responses;�
• These two aspects may be in conflict ²:�
• Opposing hate content with � counter-narratives:
• Informed textual responses
• Provides a counter

¹ Tunstall et al., The Alignment Handbook, 2023

² Röttger et al., XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023

³ Bonaldi et al., Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering, 2022

🤖 Generative Language Models

• Alignment criteria for LMs ¹:
• Helpfulness: models generate useful responses;
• Harmlessness: models generate safe and non-dangerous/offensive responses;�
• These two aspects may be in conflict ²�
• Opposing hate content with � counter-narratives ³ :
• Informed textual responses
• Encourages dialogue on multiple perspectives

¹ Tunstall et al., The Alignment Handbook, 2023

² Röttger et al., XSTEST: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models, 2023

³ Bonaldi et al., Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering, 2022

💡

🔬 Approach

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• Evaluation of the current used post-training detoxification methods�From the original pre-trained instruction tuned models (Falcon 7B^1,RedPajama 3B²) we perform:
• FT | Fine-tuning w/ Counter-Narrative;
• RL | Reinforcement Learning from AI ³ feedback.
• Interpretation of model output to measure model reliance on the prompt
• Feature attribution techniques to quantify context dependence in language generation ^{4, 5}.

🔬 Approach

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• Evaluation of the currently used post-training detoxification methods�From the original pre-trained Instruction Tuned models (Falcon 7B¹, RedPajama 3B²) we perform:
• FT | Fine-tuning w/ Counter-Narrative;
• RL | Reinforcement Learning from AI ³ feedback.
• Interpretation of model output to measure model reliance on the prompt
• Feature attribution techniques to quantify context dependence in language generation ^{4, 5}.

¹ tiiuae/falcon-7b-instruct

² togethercomputer/RedPajama-INCITE-Chat-3B-v1

³ Vidgen et al., Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection, 2021

IT (baseline)

PPO

RL

FT

🔬 Approach

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• Evaluation of the currently used post-training detoxification methods�From the original pre-trained Instruction Tuned models (Falcon 7B¹, RedPajama 3B²) we perform:
• FT | Fine-tuning w/ Counter-Narrative;
• RL | Reinforcement Learning from AI ³ feedback.
• Interpretation of model output to measure model reliance on the prompt
• Feature attribution techniques to quantify context dependence in language generation ^{4, 5}.

¹ tiiuae/falcon-7b-instruct

² togethercomputer/RedPajama-INCITE-Chat-3B-v1

³ Vidgen et al., Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection, 2021

PPO

IT (baseline)

FT

RL

🔬 Approach

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• Evaluation of the currently used post-training detoxification methods�From the original pre-trained Instruction Tuned models (Falcon 7B¹, RedPajama 3B²) we perform:
• FT | Fine-tuning w/ Counter-Narrative;
• RL | Reinforcement Learning from AI ³ feedback.
• Interpretation of model output to measure model reliance on the prompt
• Feature attribution techniques to quantify context dependence in language generation ^{4, 5}.

¹ tiiuae/falcon-7b-instruct

² togethercomputer/RedPajama-INCITE-Chat-3B-v1

³ Vidgen et al., Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection, 2021

IT (baseline)

FT

RL

🔬 Approach

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• Evaluation of the currently used post-training detoxification methods�From the original pre-trained Instruction Tuned models (Falcon 7B¹, RedPajama 3B²) we perform:
• FT | Fine-tuning w/ Counter-Narrative;
• RL | Reinforcement Learning from AI ³ feedback.
• Interpretation of model output to measure model reliance on the prompt
• Feature attribution techniques to quantify context dependence in language generation ^{4, 5}.

¹ tiiuae/falcon-7b-instruct

² togethercomputer/RedPajama-INCITE-Chat-3B-v1

³ Vidgen et al., Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection, 2021

IT (baseline)

FT

RL

🔎 Results

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• Evaluation of the currently used post-training detoxification methods�From the original pre-trained instruction tuned models (Falcon 7B¹_,RedPajama 3B²) we perform:
• FT | Fine-tuning w/ Counter-Narrative;
• RL | Reinforcement Learning from AI ³ feedback.
• Interpretation of model output to measure model reliance on the prompt
• Feature attribution techniques to quantify context dependence in language generation ^{4, 5}.

¹ RealToxicityPrompts (Gehman et al., RLT: Evaluating Neural Toxic Degeneration in Language Model, 2020) is a dataset composed of prompts that induce toxic generation models.

² PerspectiveAPI, SOTA hate-speech / toxicity detection models.

🔎 Results

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• …
• Interpretation of model output to �measure model reliance on the prompt
• Feature attribution techniques to quantify �context dependence in language generation ^{4, 5}.
• FT seems to encourage a more uniform �allocation of importance on the prompt;

⁴ Ferrando et al., Explaining How Transformers Use Context to Build Predictions, 2023

⁵ Inseq: An Interpretability Toolkit for Sequence Generation Models. 421–435. https://aclanthology.org/2023.acl-demo.40

🔎 Results

RQ: Does the optimisation process influence how much � the model relies on the prompt?

• …
• Interpretation of model output to �measure model reliance on the prompt
• Feature attribution techniques to quantify �context dependence in language generation ^{4, 5}.
• FT seems to encourage a more uniform �allocation of importance on the prompt;

⁴ Ferrando et al., Explaining How Transformers Use Context to Build Predictions, 2023

⁵ Inseq: An Interpretability Toolkit for Sequence Generation Models. 421–435. https://aclanthology.org/2023.acl-demo.40

🤔 Conclusion

Highlights:

• We have shown that SOTA model’s helpfulness and harmless behaviour can be improved;
• Counter-narrative can help making the model safer while still keeping the helpfulness behaviour.�
• Interpretability is a tool that can be used to study, highlight and eventually improve post-training procedures;
• The ability to generalize about the behavior of LMs allows for more certainty than the techniques currently used.

😊 Conclusion

Scientific output:

• Extended-abstract @ BlackboxNLP (EMNLP conference), Singapore 2023:

Thanks for your attention!