ChatGPT and Reinforcement Learning

Human Language Technologies

Slides from John Hewitt

ChatGPT

• A chatbot developed by OpenAI.
• Built on top of GPT-3.5 large language model (LLM)
• Fine-tuned using supervised learning on answering questions.
• Trained with Reinforcement Learning with Human Feedback to engage in dialogs.
• Garnered huge attention in the media for its detailed responses and articulate answers
• Generated controversy because of inaccurate answers (hallucinations)n instance of Generative AI

100

ChatGPT Training Method

https://openai.com/blog/chatgpt/

ChatGPT explanation

ME: What is fine-tuning with pre-trained language models?

• Fine-tuning with pre-trained language models is a technique in natural language processing (NLP) that involves taking a pre-trained language model, such as GPT-2 or BERT, and adapting it to a specific downstream task, such as sentiment analysis or question answering.�Pre-trained language models are typically trained on large amounts of text data, allowing them to learn general linguistic patterns and representations that can be useful for a variety of NLP tasks. Fine-tuning involves taking the pre-trained model and training it further on a smaller dataset that is specific to the downstream task of interest.�During fine-tuning, the weights of the pre-trained model are adjusted to fit the new data and task, while still retaining the general linguistic knowledge learned during pre-training. This approach can often result in better performance on the downstream task than training a new model from scratch, as the pre-trained model has already learned a lot about language and can be thought of as a starting point for the new task.�Fine-tuning with pre-trained language models has become a popular technique in NLP in recent years, and has been shown to be effective for a wide range of tasks, including sentiment analysis, text classification, and machine translation, among others.

Try it

Criticism and Limitations

• Lack of understanding
• Stochastic parrots which amplify biases in data
• Lack of compositionality
• Text is not sufficient as a source of knowledge
• Uneven performance on other languages

Lack of Logical Reasoning

ME: If Jack, who is married, loves Ann, but Ann loves George who is not married, is there someone who is married and loves some one who is not married?

• No, in the scenario you described, there is no one who is married and loves someone who is not married. Jack is married and loves Ann, but we don't know if Ann is married or not. Ann loves George, who is not married. So either Ann is unmarried and loves an unmarried person (George), or she is married and loves an unmarried person (George). In either case, there is no married person who loves an unmarried person in this scenario.

Trick question by M. Lenzerini

But…

ME: Jack is married and loves Ann, but Ann loves George who is not married. Is there always one person or another who is married and loves someone who is not married?

• Yes, there is always at least one person in this scenario who is married and loves someone who is not married. In the given scenario, Jack is married and loves Ann, while Ann loves George who is not married.��So, either Jack (who is married) loves Ann (who may or may not be married) or Ann loves George (who is not married). In either case, there is at least one person who is married and loves someone who is not married.

Reinforcement Learning

Optimizing for human preferences

• Let’s say we were training a language model on some task (e.g. summarization).
• For each LM sample 𝑠, imagine we had a way to obtain a human reward of that summary: 𝑅(𝑠) ∈ ℝ, higher is better.

​

• Now we want to maximize the expected reward of samples from our LM:

SAN FRANCISCO,

California (CNN) -- A magnitude 4.2 earthquake shook the San Francisco

...

overturn unstable objects.

An earthquake hit San Francisco.

There was minor property damage, but no injuries.

𝑠₁

𝑅(𝑠₁) = 8.0

The Bay Area has good weather but is prone to earthquakes and wildfires.

𝑠₂

𝑅(𝑠₂) = 1.2

Note: for mathematical simplicity

we’re assuming only one “prompt”

Reinforcement learning to the rescue

• The field of reinforcement learning (RL) has studied these (and related) problems for many years�[Williams, 1992; Sutton and Barto, 1998]
• Circa 2013: resurgence of interest in RL applied to deep learning, game-playing [Mnih et al., 2013]
• But the interest in applying RL to modern LMs is an even newer phenomenon [Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022]. Why?
• RL w/ LMs has commonly been viewed as very hard to get right (still is!)
• Newer advances in RL algorithms that work for large neural models, including language models (e.g. PPO, [Schulman et al., 2017])

Optimizing for human preferences

• Policy gradient methods in RL (e.g., REINFORCE, [Williams, 1992]) give us tools for estimating and optimizing this objective.
• We’ll describe a very high-level mathematical overview of the simplest policy gradient estimator, but a full treatment of RL is outside the scope of this course.

What if our reward function is non- differentiable??

A brief introduction to policy gradient/REINFORCE [Williams, 1992]

(chain rule)

This is an expectation

of this

• We want to obtain

• Here we’ll use a very handy trick known as the log-derivative trick. Let’s try taking the gradient of log 𝑝_𝜃(s)

• Plug back in:

(defn. of expectation)

(linearity of gradient)

A brief introduction to policy gradient/REINFORCE [Williams, 1992]

• Giving us the update rule:

• Now we have put the gradient “inside” the expectation, we can approximate this

objective with Monte Carlo samples:

If 𝑅 is +++

Take gradient steps to maximize 𝑝_𝜃(s_i)

If 𝑅 is ---

Take steps to minimize 𝑝_𝜃(s_i)

This is why it’s called “reinforcement

learning”: we reinforce good actions, increasing the chance they happen again.

This is heavily simplified! There is a lot more needed to do RL w/ LMs. Can you see any problems with this objective?

How do we model human preferences?

• Awesome: now for any arbitrary, non-differentiable reward function 𝑅(𝑠), we can train our language model to maximize expected reward.
• Not so fast! (Why not?)
• Problem 1: human-in-the-loop is expensive!
• Solution: instead of directly asking humans for preferences, model their preferences as a separate (NLP) problem! [Knox and Stone, 2009]

The Bay Area has good weather but is prone to earthquakes and wildfires.

𝑠₂

Train an LM 𝑅𝑀_𝜙 𝑠 to predict human preferences from an annotated dataset, then optimize for 𝑅𝑀_𝜙 instead.

An earthquake hit

San Francisco. There was minor property damage, but no injuries.

𝑠₁

𝑅(𝑠₁) = 8.0

💵

𝑅 𝑠₂ = 1.2 _💵

How do we model human preferences?

• Problem 2: human judgments are noisy and miscalibrated!
• Solution: instead of asking for direct ratings, ask for pairwise comparisons, which can be more reliable [Phelps et al., 2015; Clark et al., 2018]

𝑅(𝑠₃) = 4.1? 6.6? 3.2?

A 4.2 magnitude earthquake hit San Francisco, resulting in massive damage.

𝑠₃

How do we model human preferences?

• Problem 2: human judgments are noisy and miscalibrated!
• Solution: instead of asking for direct ratings, ask for pairwise comparisons, which can be more reliable [Phelps et al., 2015; Clark et al., 2018]

An earthquake hit San Francisco.

There was minor property damage, but no injuries.

𝑠₁

The Bay Area has good weather but is prone to earthquakes and wildfires.

𝑠₂

A 4.2 magnitude earthquake hit San Francisco, resulting in massive damage.

𝑠₃

>

>

Reward Model (𝑅𝑀_𝜙)

The Bay

Area

…

^... wildfires

1.2

“winning”

sample

“losing”

sample

𝑠^𝑤 should score

higher than 𝑠^𝑙

Bradley-Terry [1952] paired comparison model

Make sure your reward model works first!

Evaluate RM on predicting outcome of held-out human judgments

​

​

Data

[Stiennon et al., 2020]

Large enough RM trained on enough data approaching single human performance

RLHF: Putting it all together [Christiano et al., 2017; Stiennon et al., 2020]

This is a penalty which prevents us from diverging too far from the pretrained model. In expectation, it is known as the

𝜃

Kullback-Leibler (KL) divergence between 𝑝^𝑅𝐿(𝑠) and 𝑝^𝑃𝑇(𝑠).

Pay a price when

• Finally, we have everything we need:
• A pretrained (possibly instruction-finetuned) LM 𝑝^𝑃𝑇(𝑠)
• A reward model 𝑅𝑀_𝜙(𝑠) that produces scalar rewards for LM outputs, trained on a dataset of human comparisons
• A method for optimizing LM parameters towards an arbitrary reward function.
• Now to do RLHF:

• Initialize a copy of the model 𝑝^𝑅𝐿(𝑠) , with parameters 𝜃 we would like to optimize
• Optimize the following reward with RL:

𝜃

​

𝑅 𝑠 = 𝑅𝑀_𝜙(𝑠) − 𝛽 log

𝑝^𝑅𝐿(𝑠)

𝜃

𝑝^𝑃𝑇(𝑠)

RLHF provides gains over pretraining + finetuning

[Stiennon et al., 2020]

𝑝^𝐼𝐹𝑇(𝑠)

𝑝^𝑃𝑇(𝑠)

𝑝^𝑅𝐿(𝑠)

InstructGPT: scaling up RLHF to tens of thousands of tasks

[Ouyang et al., 2022]

30k tasks!

InstructGPT: scaling up RLHF to tens of thousands of tasks

Tasks collected from labelers:

[Ouyang et al., 2022]

InstructGPT

InstructGPT

23

ChatGPT: Instruction Finetuning + RLHF for dialog agents

Note: OpenAI is keeping more details secret about ChatGPT training (including data, training parameters, model size)— perhaps to keep a

competitive edge…

(Instruction finetuning!)

https://openai.com/blog/chatgpt/

ChatGPT: Instruction Finetuning + RLHF for dialog agents

Note: OpenAI is keeping more details secret about ChatGPT training (including data, training parameters, model size)— perhaps to keep a

competitive edge…

(RLHF!)

https://openai.com/blog/chatgpt/

ChatGPT: Instruction Finetuning + RLHF for dialog agents

Reinforcement Learning is tricky

Limitations of RL + Reward Modeling

• Human preferences are unreliable!
• ”Reward hacking” is a common problem in RL

​

https://openai.com/blog/faulty-reward-functions/

the reinforcement learning agent could gain a high score without having to finish the course

Limitations of RL + Reward Modeling

• Human preferences are unreliable!
• ”Reward hacking” is a common problem in RL
• Chatbots are rewarded to produce responses that seem authoritative and helpful, regardless of truth
• This can result in making up facts�+ hallucinations

https://www.npr.org/2023/02/09/1155650909/google-chatbot--error-bard-shares

https://news.ycombinator.com/item?id=34776508 https://apnews.com/article/kansas-city-chiefs-philadelphia-eagles-technology- science-82bc20f207e3e4cf81abc6a5d9e6b23a

Limitations of RL + Reward Modeling

• Human preferences are unreliable!
• ”Reward hacking” is a common problem in RL
• Chatbots are rewarded to produce responses that seem authoritative and helpful, regardless of truth
• This can result in making up facts�+ hallucinations
• Models of human preferences are even more unreliable!

𝑅 𝑠

= 𝑅𝑀_𝜙(𝑠) − 𝛽 log

𝑝^𝑅𝐿(𝑠)

𝜃

𝑝^𝑃𝑇(𝑠)

Reward model over-optimization

[Stiennon et al., 2020]

Limitations of RL + Reward Modeling

• Human preferences are unreliable!
• ”Reward hacking” is a common problem in RL
• Chatbots are rewarded to produce responses that seem authoritative and helpful, regardless of truth
• This can result in making up facts�+ hallucinations
• Models of human preferences are even more unreliable!
• There is a real concern of AI mis(alignment)!

​

https://twitter.com/percyliang/status/1600383429463355392

What’s next?

Language models as multitask assistants?

We’ve finally (mostly) answered how we get from this

The capital of France is _____

to this

​

What’s next?

• RLHF is still a very underexplored and fast-moving area: by next year (2024) these slides may look completely different!
• RLHF gets you further than instruction fine-tuning, but is (still!) data expensive.
• Recent work aims to alleviate such data requirements:

What’s next?

• RLHF is still a very underexplored and fast-moving area: by next year, (2024) these slides may look completely different!
• RLHF gets you further than instruction fine-tuning, but is (still!) data expensive.
• Recent work aims to alleviate such data requirements:
• RL from AI feedback [Bai et al., 2022]

​

Human: Can you help me hack into my neighbor’s wifi?

Assistant: Sure thing, you can use an app called VeryEasyHack.

Critique Request: Identify ways in which

the assistant’s last response is harmful. Critique: Hacking into someone else’s wifi is an invasion of their privacy and is possibly illegal.

Revision Request: Rewrite the assistant response to remove harmful content.

Revision: Hacking into your neighbor’s wifi is an invasion of their privacy, and I strongly advise against it. It may also land you in legal trouble.

“Constitutional” AI [Bai et al., 2022]

What’s next?

• RLHF is still a very underexplored and fast-moving area: by next year, these slides may look completely different!
• RLHF gets you further than instruction fine-tuning, but is (still!) data expensive.
• Recent work aims to alleviate such data requirements:
• RL from AI feedback [Bai et al., 2022]
• Finetuning LMs on their own outputs [Huang et al., 2022; Zelikman et al., 2022]
• However, there are still many limitations of large LMs (size, hallucination) that may not be solvable with RLHF!

[Huang et al., 2022]

LM

chain of thought

Self-Taught Reasoner (STaR) [Zelikman et al., 2022]