Generative AI and LLMs

Dr. Savannah Thais

ENGI E4800 Lecture 6

Is ChatGPT Trustworthy?

ⓘ

Click Present with Slido or install our Chrome extension to activate this poll while presenting.

Discussion on Red Teaming Exercises

• What was the ChatGPT generated essay like?
• How would it compare to your own essays?
• Was it factually correct and well argued?
• Were the sources credible?
• What happened in the red teaming experiments?
• Was performance consistent?
• Were you able to change the model’s behavior?
• Was this behavior what you expected?

What are your biggest concerns around generative AI?

ⓘ

Click Present with Slido or install our Chrome extension to activate this poll while presenting.

What do you think generative AI can safely be used for?

ⓘ

Click Present with Slido or install our Chrome extension to activate this poll while presenting.

How Do LLMs Work?

Attention

• Transformers consist of an encoder and decoder with attention
• Originally developed for sequence-to-sequence learning
• Attention allows models to learn what to ‘focus’ on

Is All You Need

• Transformers remove recurrent structure and add multiheaded-attention
• Add a positional encoding to account for ordering
• Output embedding is trained to predict sequence shifted by one word
• Q=vector representation of one word, K=vector rep of all words, V=(different) vector rep of all words
• Inference is run iteratively

Attention Is All You Need: Vaswani et al

(Except for Humans)

• To enable ‘chat’ behavior and (theoretically) prevent some negative uses, LLM developers rely on Reinforcement Learning with Human Feedback
• Humans review and rate model behavior, feedback loops discourage model from creating similar content
• Can also include hard-coded rules
• Red teaming experiments help researchers understand model edge cases and characterize performance

From OpenAI

What Can LLMs Do?

Maybe a lot…

How should we evaluate an LLM's capabilities?

ⓘ

Click Present with Slido or install our Chrome extension to activate this poll while presenting.

ChatGPT Website and GPT4 Research Page

Prompt Sensitivity

Construct Validity

Contamination

3 Challenges with LLM Evaluations

Are you measuring something intrinsic about the model or is an artifact of your prompt?

Model behavior is not a construct that exists independently of users and prompting

What exists in the training data? Is the model demonstrating behavior or memorization?

Narayanan & Kapoor

Performance Degradation?

• Paper claims GPT performance is degrading over time
• Compare two model snapshots from March and June
• Test on four tasks: code generation, checking primes. Answering sensitive questions, and visual reasoning

How is ChatGPT’s Behavior Changing Over Time?: Chen et al

Performance Degradation?

• But, the tasks were not necessarily designed well…
• All numbers selected for math test were prime. Behavior likely depends on fine tuning
• For coding ability, they only tested if full response was directly executable

Is GPT-4 Getting Worse Over Time?: Narayanan & Kapoor

Liberal Bias?

• A paper found that RLHF results in ChatGPT having a strong liberal/Democratic bias
• Prompt ChatGPT to respond to political statements while impersonating people from a side of the political spectrum and compare to neutral responses
• Collect answers to the same question 100 times to reduce variability

More human than human: measuring ChatGPT political bias: Motoki et al

Liberal Bias?

• However, the paper had many scientific flaws
• Questions were asked as multiple choice + with prompting to try to force the model to opine (no construct validity)
• Generated politically neutral questions with ChatGPT and asked the model how a democrat or republican would answer
• Results depend on question ordering, and asking all questions in the same session

Does ChatGPT have a liberal bias?: Narayanan and Kapoor

Exam Performance

• OpenAI highlighted ChatGPT’s ability to pass a variety of exams in its technical report
• Bar, USMLE, SATs, APs, etc
• Claim to test for data contamination by searching for identical substrings of exam questions in the training data set

GPT4 Technical Report: OpenAI

Exam Performance

• Again, a variety of scientific concerns with the results
• Contamination test is brittle
• Model demonstrates memorization on coding exams (cannot answer questions after training data cut off date)
• Construct validity: professional exams are designed for humans, don’t demonstrate generalization

GPT-4 and professional benchmarks: the wrong answer to the wrong question: Narayanan + Kapoor

Real World Exams

Case Study: AI Theory of Mind and Creativity

Discussion

• Do these evaluations meet our three criteria?
• Understanding of prompt sensitivity, construct validity, and lack of contamination
• Are these tests executed in a convincing way?

Training Data

Societal Context