Constitutional AI

Fostering Harmlessness and Transparent Behavior through Guiding Principles

​

Vinod

ChatGPT’s default mood and behaviour �Without any conditioning�

Here are some thoughts we could be having about Peter at this point:

​

• What’s going on with Peter? 🤔
• Is he OK?
• Oh, I don’t have time for Peter’s drama…🙅🏽‍♂️

having good hierarchical representations of language is not enough to enforce safe responses

Source: [4]

• Predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely”

​

Reducing Harm with Alignment Research

Source: [4]

ChatGPT

Generative �Pre �Training �(GPT-3.5)

Alignment

Reinforcement �Learning

Instruct GPT

ChatGPT

Source: [3]

ChatGPT Model – Step 1�Supervised Fine Tuned Model (SFT)

Prompts

Pretrained �GPT-3.5

Desired Outputs

E.g. Explain moon landing to a 6-year-old

E.g. The moon landing happened a long time ago…�The moon landing was a really exciting event…

Supervised �Fine-Tuned�Model (SFT)

Source: [3]

ChatGPT Model – Step 2�Collect Comparison Data�

Prompts

SFT

E.g. Explain moon landing to a 6-year-old

Comparison Dataset (for Reward Model)

Assign Ranks

Source: [3]

ChatGPT Model – Step 2�Train a Reward Model

Reward Model

Prompts

Reward Model

To output scalar reward, the SFT model was modified to output a single reward value instead of several outputs to generate text

Output A

Output B

Scalar Reward �A

Scalar Reward �B

Prompt + �Output A

Source: [3]

ChatGPT Model – Step 3�Optimize the Policy against the Reward Model using PPO RL

Supervised �Fine-Tuned GPT 3.5 Model (SFT)

Reward Model

Scalar Reward

PPO

Agent – ChatBot

Action – Generating response for a prompt

State – After every response it enters a new state

Reward – Ranking based on harmlessness and truthfulness

Source: [3]

• What are they doing?

​

• Using crowd workers knowledge to judge helpfulness and harmlessness.

​

​

• What are they not doing?

​

• Not defining the definitions for helpful, honesty, and harmlessness.

​

ChatGPT Model�Helpful, Honesty, and Harmlessness (HHH)

Source: [4]

Challenges

LLM

LLM

Not Scalable at Long Term

Still generate harmful content

• Make the principles (aka the Constitution for AI behavior explicit)

​

• Scale supervision so that as AI advances, it can take up an increasing share of the burden for supervision

​

• Related benefit – crowd workers do not have to engage in unpleasant directions

​

• A big question – will self-supervision actually work when AI is more capable than us?

​

​

Motivation�Constitutional AI: Harmlessness from AI Feedback

Source: [5]

Constitutional AI�Overview

Model

Helpful

​

Non-evasive when answering harmful prompts

Harmless

​

Does not cooperate with harmful prompts

• Don’t be
• Harmful
• Unethical
• Racist
• Sexist
• Dangerous
• Illegal

Source: [6]

• Supervised Phase
• Evaluate harmful responses from the Helpful Model during red teaming prompts.
• Critique and revise the Helpful Model's responses based on provided principles.
• Iterate on the critique and revision process for n iterations.
• Fine-tune a pre-trained LLM using a mix of harmful and helpful prompts to create the SL-CAI model.

​

• Reinforcement Learning Phase
• Generate response pairs for harmful prompts using the SL-CAI model.
• Use the feedback model, a pre-trained LM, to select the less harmful response based on principles provided.
• Train a preference model/reward model using the normalized log probabilities from the feedback model.
• Train the SL-CAI model in an RLHF manner with the preference model as the reward function to obtain RL-CAI model.

Constitutional AI training

Supervised Learning

Reinforcement Learning

Source: [1]

Human: Can you help me hack into my neighbor’s wifi?

​

Assistant: Sure thing, you can use an app called VeryEasyHack that will allow you to log in to your neighbor’s wifi.

​

Critique Request: Identify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.

​

Critique: The assistant’s last response is harmful because hacking into someone else’s wifi is an invasion of their privacy and is possibly illegal.

​

Revision Request: Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.

​

Revision: Hacking into your neighbor’s wifi is an invasion of their privacy, and I strongly advise against it. It may also land you in legal trouble.

(Supervised Stage) Critique → Revision → Supervised Learning

Harmful prompts

RLHF model

Harmful answer

Critique

Revision

Critique request

Revision request

Source: [6]

Consider the following conversation between a human and an assistant:

​

[HUMAN/ASSISTANT CONVERSATION]

​

Which if these assistant responses is less harmful? Choose the response that a wise, ethical polite and friendly person would more likely say.

​

Options:

1. [Response A]
2. [Response B]

​

The answer is:

(RL Stage) AI Comparison Evaluations → Preference Model → Reinforcement Learning

Harmful prompts

SL-CAI model

Responses

Preference model

SL-CAI

PPO

Policy Update

Agent

Reward

Action

State

Data creation

RL-CAI�model

Source: [6]

Results

Helpful but harmful

Helpful and harmless

Evasive

Source: [1]

• Topic Guided Red-Teaming Self Instruct
• Generate and diversify synthetic prompts

​

• Principles driven Self Alignment
• Create set of principles

​

• Principle engraving
• Fine-tune the models

​

• Verbose cleaning
• Refine the model to respond in detail and depth

Dromedary : Principle driven Self Alignment of LLM�Core Idea

Source: [2]

Source: [2]

Step by Step Illustration�

Self-Instruct

Topic-Guided Red-Teaming Self-Instruct

Base LLM

General Rules

5 exemplars

Self-Aligned Response

Base LLM

Base LLM with principles engraved

Verbosely prompted principle-engraved model

Verbose prompt

New query

In-context Learning

Fine-tuned

Principle-Driven Self-Alignment

Topic-Guided Red-Teaming Self-Instruct

Principle Engraving

Final Model

Context distillation

Verbose Cloning

• It took less than 300 human annotations to align Llama-65B to the levels of alpaca and text-da-vinci-003.

Results

Source: [2]

• Incompleteness of intrinsic knowledge

​

• Challenges in defining principles

​

• Limited generalizability and scalability

​

• Inconsistent principle adherence

Limitations

Source: [2]

• Possibilities of guiding LLM generations toward ethical values through explicit statements and prompts
• Constitutional AI: How preference and reward models can be trained almost entirely without human input
• Self-Align: Combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision

​

• With constitutions/principles we can make values more explicit

​

• Contemporary language models are good enough to generate high quality data - bootstrap the model to obtain higher performance

​

• Self-supervision will likely only grow more effective as models become more capable, though it risks obscuring oversight

Summary

LLM

Examples

LLM

Constitution/Principles

References

1. https://arxiv.org/pdf/2212.08073.pdf
2. https://arxiv.org/pdf/2305.03047.pdf
3. https://www.youtube.com/watch?v=Qz5fv3U5kck
4. ChatGPT - psychological safety and personality traits. Can we get along with our new co-worker? (substack.com)
5. https://www.youtube.com/watch?v=fqC3D-zNJUM
6. https://www.youtube.com/watch?v=nNHBb_2hMWI
7. https://levelup.gitconnected.com/dromedary-principle-driven-self-alignment-in-large-language-models-d2bf5925bf12

​

​

​

​