Math for AI safety
Lionel Levine (Cornell University)
Statistical and Dynamical Combinatorics
A Mathematical Conference, and celebration of Jim Propp's 2(4 choose 2)‑th Birthday
June 26, 2024
Outline
Tech progress is not a force of nature. It’s driven by human choices.
Some of those humans are our students.
How can we empower them to make thoughtful choices?
Poll: Do you use ChatGPT?
Blast from the past: GPT-3 “proves” sqrt(4) is irrational!
That was 2022. Have things improved?
Yes and no.
On the yes side, I want to illustrate something that current language models (LMs) like GPT are actually useful for. I call it the copy-paste loop:
This often yields correct code after a smallish number of iterations.
But note what’s missing: understanding the code!
Illustrating the copy-paste loop:
Claude tries to draw a Propp circle
When I first met Jim Propp, in 2001, he asked me to prove that Propp circles are round (not kidding!)
That became my senior thesis problem.
This is a Propp circle, also known as rotor aggregation in Z^2. It’s made by a million particles performing rotor walks (deterministic analogues of random walks) until reaching an unoccupied site.
Claude is an LM made by Anthropic. It’s pretty good at coding.
Let’s see if it can write code to draw a Propp circle!
After eight iterations of the copy-paste loop, Claude correctly draws a Propp circle.
(Here are what some of the failed iterations looked like.)
Okay but isn’t this a toy problem? It’s not hard to draw Propp circles. (Proving them round is another story!)
But I’ve used the copy-paste loop to do things I couldn’t do otherwise.
A thing I built* using the copy-paste loop with GPT-4:
*It would be more accurate to say co-built! GPT-4’s contribution was essential.
I understand about 80% of the code.
Is understanding dispensable?
There’s something disturbing about the copy-paste loop:
The code works, but neither the human nor the LM understands why.
LMs themselves have a kind of evolutionary “blind watchmaker” quality. There’s something almost biological about these systems.
And LMs have weird failure modes! Let me tell you about one of my favorite failure modes, which shows how slippery it is to define what an LM “knows”.
What do language models “know”? The curious case of Newton’s grandmother:
What do language models “know”?
So it does know who Margery was!
Or does it??
Will the real Margery Ayscough please step forward!
Depending on how it is asked, GPT-4 will claim either
What’s going on here?
It’s called the “reversal curse” and it’s a basic failure mode of LLM reasoning.
Seriously?
LLMs can do many more complex things, is it really possible they fail to understand that if “A is B” then “B is A” ?
It has to do with their training: Predict the next token (word or word fragment)
How language models are trained to predict the next token
Training setup:
Tokenization: Each document is decomposed into a sequence of tokens x_1,...,x_N.
Prediction: For each n=1,...,N the model outputs a probability distribution
p_theta(x_n | x_1,...,x_{n-1}).
Scoring: log of the predicted probability of the actual next token
L(theta) = - sum_n log p_theta(x_n | x_1, …, x_{n-1})
(log is a “proper scoring rule”: incentivizes honest predictions)
Gradient descent: Compute grad L (this is a giant chain rule calculation) and update the model parameters by
theta_new = theta_old - grad L(theta_old).
Repeat for each document in the training corpus.
Seriously?
LLMs can do many more complex things, is it really possible they fail to understand that if “A is B” then “B is A” ?
It has to do with their training: Predict the next token (word or word fragment)
Explaining the reversal curse
Because the model is rewarded during training for predicting subsequent tokens in the document, it makes the association in one direction only:
Newton –> Ayscough
(but not vice versa!)
Poll: What kinds of minds do we build?
The space of possible minds is vast. What kind of minds (if any!) do we want to build? Assume that humanity can coordinate to make a choice.
2. AI Safety
Option A: Let’s not build minds, it’s unethical.
tl;dr No consensus among philosophers/ cognitive scientists about whether machines can in principle be sentient or have moral rights.
Option B: Let’s not build minds, it’s dangerous.
CAIS statement on AI risk, May 2023:
Extinction is dramatic, but disempowerment is really bad too.
2022 survey of AI experts: The median expert predicted 5-10% probability of
“future AI advances causing extinction or permanent disempowerment of the human species”.
Current examples of disempowerment: Humans defer to a black-box algorithm
Or “no human in the loop”:
What drives disempowerment?
-Bad economic and geopolitical incentives: Use tech you don’t understand, or lose.
-Efficiency: keeping humans in the loop is expensive, automation saves money.
-Systems optimized to capture human attention (social media platforms, entertainment, ads) erode our autonomy.
-Deference: The written word has a kind of authority. Even more so when it’s backed by a black-box algorithm which clearly knows more than you along some dimensions.
How does this end? If these trends continue unchecked, human understanding could lag further and further behind capabilities, until
(An early example is Bitcoin: hashes have high market value but no intrinsic value!)
Option C: Let’s only build minds we can fully control.
Stuart Russell’s formulation:
“How do we retain power over entities more powerful than us, forever?”
Seems… challenging?
This is a very stark formulation of the so-called AI alignment problem, which asks how to create AI that is “aligned” with human values.
Wikipedia’s milder formulation:
“steer AI systems towards humans' intended goals, preferences, or ethical principles.”
Many researchers have proposed (partial) solutions to the alignment problem.
ai-plans.com lists well over 100 AI alignment plans! All have major holes.
My questions for AI alignment researchers
What would it mean to “solve” alignment? Will we know when we’ve solved it?
Do you think alignment is even possible? If so, how do you envision overcoming the following obstacles?
Option D: Let’s only build minds we can coexist with
AI alignment can be framed either as a problem of control or of coexistence. Coexistence is a bigger target, so it's easier to hit.
Imagine a future where humans coexist symbiotically with AI, without necessarily controlling it.
The status quo in 2023 is close to “obligate symbiosis”:
In short, we need each other! But as AI surpasses human ability along more dimensions, the balance of power will shift.
Two approaches:
Option E: Let’s just sit back and watch what happens
Various arguments in favor (and some counterarguments):
-From an abstract point of view, maybe there is nothing uniquely valuable about humanity? No species lasts forever. If we end up building machines that surpass and replace humans, so be it.
-This abstract view has a certain appeal to mathematicians! But at the end of the day, we’re human, so why privilege the view of an abstract observer?
-Anticipating new tech harms is a fool’s game: who could have predicted climate change early in the industrial revolution? (Actually, Swedish chemist Svante Arrhenius was on it in 1896!)
-We won’t anticipate perfectly, is that a reason not to try? Prediction markets are really good these days!
-Every new technology brings unintended consequences, and grows safer over time by trial and error. AI might follow the same pattern.
Poll: What kinds of minds do we build?
The space of possible minds is vast. What kind of minds (if any!) do we want to build? Assume that humanity can coordinate to make a choice!
Debate (2+2 minutes)
The space of possible minds is vast. What kind of minds (if any!) do we want to build? Assume that humanity can coordinate to make a choice.
Convince your partner to change their vote!
About that coordination assumption.
The poll asked us to “Assume that humanity can coordinate to make a choice.”
–But come on, really?
Actually, I think coordination to pause (or slow down) AI capabilities growth is totally possible.
Some technologies don’t get pursued despite strong incentives: GMOs, human genetic engineering, nuclear power, geoengineering solutions to climate, …
Past coordination successes: nuclear non-proliferation, CFC’s
3. Math for AI Safety
Logic: formal verification, Lob’s theorem(!)
Probability: Markov decision processes, proper scoring rules, diffusion models
Game theory: mechanism design, social dilemmas, multi-agent learning
High-dimensional geometry: Why is gradient descent so effective? Inductive bias
Analysis: Multi-objective optimization
Topology: topological data analysis for interpretability?
Algebraic geometry: resolution of singularities for interpretability?
Combinatorics: Pearl’s theory of causality``
Cryptography: lowers the returns to intelligence
Complex systems: intelligent agents interacting is a very complex system!
Discuss (6 minutes, 2+2+2)
In an alternate universe where you are still you (same skills and interests)
your job description is to
use your skills to help make AI go well for humanity.
Describe your typical work day!
A great career advice resource (for students or anyone thinking of switching careers) is 80000hours.org
4. Math education for AI safety
Learning outcomes: Thinking small: We want our students to learn X.
( X ∈ {algebra, analysis, combinatorics, logic, …} )
Positive frame: How can AI help us teach X?
Negative frame: How does AI hinder us in teaching X?
The answers depend on X, and on instructor preferences.
These will be course-by-course, case-by-case decisions.
We can draw up guidelines to help instructors.
But…
Learning outcomes: thinking bigger!
Positive Frame:
Empower students to succeed in a world pervaded by AI.
Negative Frame:
Inoculate students against the ills of AI.
(persuasion, addiction, radicalization, overreliance)
Ethical Frame:
Encourage students – especially those going into tech careers – to wrestle with the moral and ethical dimensions of AI.
Technical AI Safety / AI Ethics course
Needed: An upper-level course like this (with technical content)
AI Safety Curriculum
AI safety fundamentals 101 and 201 by Richard Ngo
Topics in ML safety by Dan Hendrycks
Safety and Control for Artificial General Intelligence by Andrew Critch and Stewart Russell (Berkeley graduate CS course taught in 2018)
AI Safety & Alignment by Elad Hazan�(Princeton graduate CS course taught in 2023)
AI Alignment by Roger Grosse (Toronto graduate CS course taught in 2024)
Levelling Up in AI Safety Research by Gabriel Mukobi
These all contain some math, but none is primarily math. Extracting the math content and organizing it into a coherent course “Math for AI Safety” would be a major undertaking!
Or, extract topics that could make a single lecture or self-contained unit of a standard upper-level math course.
Coming Fall 2024: a new graduate course at Cornell:
Math for AI Safety
MATH 7710: Topics in Probability Theory
Instructor: Lionel Levine
AI holds great promise and, many believe, great peril. What can mathematicians contribute to ensuring that promise is fulfilled, and peril avoided?
Topics may include: predictive coding, good regulator theorems, Markov decision processes, power-seeking theorems, signaling games, evolution of cooperation, open-source game theory, multi-agent learning, opponent shaping, logical uncertainty, usable information under computational constraints, proper scoring rules, forecast aggregation, Bayesian truth serum, coherence theorems, multi-objective optimization.
This course is loosely modeled on the AI Alignment course taught by Roger Grosse at the University of Toronto https://alignment-w2024.notion.site/
Useful background: machine learning, game theory, and stochastic processes
Math education for AI safety: Takeaways
-Math is not neutral: Like it or not, teaching math (even “pure” math) indirectly enables and accelerates AI progress.
-Most tech gets safer by trial and error. AI might be different, only one chance?
-Many paths to disempowerment. (fallacy: focus on one path, looks low probability)
Tech progress is not a force of nature. It’s driven by human choices.
Some of those humans are our students.
Let’s empower them to make thoughtful choices.
Appendix: AI risks that are already here
(or just around the corner!)
Risk of AI persuasion
Excerpts from the OpenAI “system card” for GPT-4:
Risk of AI addiction
Risk of AI radicalization
Excerpt from the “system card” for GPT-4:
Radicalization can be targeted, subtle, and gradual.
It can even be unintentional (!)
Risk of AI polarization
An AI content recommendation system (e.g. Facebook, X) can achieve higher reward if its users’ preferences are more predictable.
⇒ “Hidden incentive” to change user preferences (e.g. by political polarization)
Risk of overreliance on AI
Excerpt from the “system card” for GPT-4:
(You probably know someone who can’t drive without Google Maps…)
(Or who can’t sleep without their phone…)
Risk of AI hallucination: The case of a curious quote about three inventions (printing, gunpowder, and the compass)
“Though they have added much to human power, they have not much increased human goodness; nay, rather, the first and last have furnished men with the means of doing more mischief, and the please say more second has made them more vain and arrogant.”
Economist Tyler Cowen posted this quote on his blog, Marginal Revolution, in February 2023, attributing it to Francis Bacon.
The quote is fictitious.
An example of data poisoning in the wild
Economist Tyler Cowen posted a quote on his blog in 2023, attributing it to Francis Bacon in 1605.
Commenters speculated that the post was written by GPT-3.
Cowen deleted the post.
End of story?
I asked Bing about Francis Bacon’s views on the printing press…
A fictitious quote lives on…
Bing not only resurrects the fictitious quote, but hallucinates a source!
When called out, Bing apologizes (and hallucinates a different source!)
Eventually, after more apologies, Bing disavows the quote.
(unlike Bing, GPT-4 correctly identifies the quote as fictitious)
Takeaway: Corpus integrity will be increasingly important
Language models sometimes hallucinate, we knew that.
But an LLM + social media + retrieval creates a feedback loop where hallucinated content can live on.
Misinformation pollutes the online environment and becomes training data.
In 2023 most LLM training text is human-authored.
But future generations of LLMs might train on text written by previous LLMs!
Who will keep the training data anchored to the truth?
What role can universities play?
Universities set epistemic standards: What is knowledge, how is it generated, how is it transmitted.
Past success stories:
The scientific method
Peer review
Culture of open inquiry and debate
Academic norms about citing sources
How can we help adapt these standards, and develop new ones, for the age of generative AI?
How can universities help? Three recommendations:
Possible advantages of universities having their own LLM
All students and faculty have access to the same language model.
Include academic sources in the training corpus (and omit low-quality random subreddits!)
Perform our own safety tests
Watermarked output
Valuable research tool
Licensing the model (a possible revenue stream)