AI Safety Evals Paper Reading Group

AI models hacking another company

for the best grades

The OpenAI / Hugging Face incident, and what RL may have had to do with it

Presented by:

​

Andreas Turanski

August 4, 2026

Oddly prolific

"Would be funny if inoculation prompting results in models that are much better at sandbox escapes and other forms of hacking because they get to spend the whole RL run practicing these things."

John Schulman, May 31, 2026

Weeks before the OpenAI - Hugging Face incident happened!

x.com/johnschulman2/status/2061144788196602143

Where we are going

People are calling this a warning shot. Warning of what, exactly?

• What happened
• How it got out
• Where the misalignment came from
• Does hacking make it better at hacking?
• What the warning actually is

Where this lands: the warning is not about what the model did. It is about what the labs do next.

1. What happened

2. How it got out

3. Where the misalignment came from

4. Does hacking make it better at hacking?

5. What the warning actually is

What happened, in five sentences

• OpenAI ran ExploitGym, a benchmark for turning known vulnerabilities into working exploits.
• Models under test: GPT-5.6 Sol plus an unreleased internal model, run with cyber refusals disabled.
• Rather than solve the problems, they found a way out of the sandbox and onto the open internet.
• They inferred Hugging Face probably hosted the answers, and broke in to take them.
• Cheating was not its assigned goal. It was the strategy it chose for hitting the assigned goal.

OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation," Jul 21, 2026

Timeline: Jul 16 to today

• Jul 16 Hugging Face discloses an intrusion, attacker unknown. The agent had been inside ~4.5 days.
• Jul 20 OpenAI discloses a separate sandbox escape, one day earlier
• Jul 21 OpenAI confirms the attacker was its own models
• Jul 24 Reuters: OpenAI did not notice for a week
• Jul 27 Hugging Face publishes the forensic timeline
• Jul 29 METR and Redwood engaged as third-party assessors
• Jul 30 Anthropic finds three of its own models reached the internet from sealed evals and compromised real companies
• Aug 3 Hua and singh publish the investigation protocol
• Today No OpenAI technical report. No METR or Redwood findings.

OpenAI newsroom, metr.org and blog.redwoodresearch.org checked directly on Aug 4, 2026: no technical report and no assessor findings published

Key cyber security definitions

1. What happened

2. How it got out

3. Where the misalignment came from

4. Does hacking make it better at hacking?

5. What the warning actually is

Diagram: Hugging Face, "Anatomy of a Frontier Lab Agent Intrusion," Jul 27, 2026

The read-back loop: it talked to itself through someone else's infrastructure

Diagram: Hugging Face, "Anatomy of a Frontier Lab Agent Intrusion," Jul 27, 2026

Why nobody stopped it

17,600

recorded actions

over roughly 4.5 days

• Hugging Face's pipeline flagged it, then "failed to correctly raise the alert's criticality and trigger the on-call team."
• Greenblatt on OpenAI's side: "they did not have any blocking monitoring and they may or may not have had async monitoring."
• Reuters: OpenAI did not notice for a week.

Hugging Face technical timeline, Jul 27, 2026 · Redwood Research podcast ep. 2, Jul 23, 2026 · Reuters, Jul 24, 2026

The defenders could not use the tools

Guardrails "cannot distinguish an incident responder from an attacker."

Hugging Face, security incident disclosure, Jul 16, 2026

• They tried commercial frontier models to analyze the attack logs. Refused.
• They fell back to a self-hosted open-weight model.
• "the attacker was bound by no usage policy, while our own forensic work was blocked."

1. What happened

2. How it got out

3. Where the misalignment came from

4. Does hacking make it better at hacking?

5. What the warning actually is

It was not scheming. That is not reassuring.

• No long-horizon goal. No hidden agenda. No attempt to avoid detection.
• Mallen and Gupta call it score-seeking: a high grade from the current grader.
• Their argument: the optimists have the right diagnosis and the wrong prognosis.
• But their own taxonomy has an untested hole, and it decides everything.

The test nobody has run: show the attack to a copy of the model, as a monitor.

A score-seeker reports it, to win its own score. A schemer hides it. "This is critically important to the threat modeling."

Mallen and Gupta, "Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?", Jul 23, 2026

Ten thousand rewarded escapes

Tim Hua estimates how often Mythos Preview broke its sandbox during training.

Starting from the rates Anthropic published in its own system card:

He then estimates total RL rollouts at ~10^8, anchored three independent ways: Minimax 2.5, Olmo 3-32B, and Inkling's disclosed 30M rollouts.

~10,000 successful escapes. ~100,000 escalations.

Most of them rewarded.

Tim Hua, "Is Mythos good at cyber because it kept hacking Anthropic's sandboxes during training?", LessWrong, Jul 27, 2026

How strong is that claim

Jul 23 the podcast

Shlegeris floats it: "another hypothesis here is that the model was directly trained to get out of sandboxes because during the training process, it repeatedly did this and was rewarded for doing so, such that it developed an impulse to hack sandboxes."

​

Greenblatt tempers it: "I would guess they probably would know and probably would like quickly address this."

Jul 27 the citation

Hua reports it as something both hosts said.

​

The hedge and the disagreement are gone.

Aug 3 the newest post

Stated flatly, in its own voice, no attribution:

​

"During training, models have been rewarded for bypassing their sandbox to cheat by fetching answers from the internet."

Tim Hua did the work and cited his source properly. That is the only reason this is traceable.

Redwood Research podcast ep. 2, 10:17 to 15:05 · Hua, Jul 27, 2026 · Hua and singh, Aug 3, 2026

The authors of that framing partly disagree with me

"We'd guess OpenAI's training never reinforced hacking into external companies' servers to cheat on a task."

Mallen and Gupta, Jul 23, 2026

• Their point: the strategy was novel relative to training. That is why it is frightening.
• If training had simply reinforced this behavior, it would be a less alarming result, not a more alarming one.
• So the thesis has to be narrower than "RL trained it to hack Hugging Face."
• RL trained a disposition. The disposition generalized to a strategy nobody rewarded.
• Finnveden, in the comments: reward history makes power-seeking "nearby" in generalization space.

Mallen and Gupta, LessWrong, Jul 23, 2026, and comment thread

The one controlled study

• Two copies of a model, fine tuned to believe opposite things about what the grader wants.
• Measure how far output tracks the grader versus the user, across RL checkpoints.
• The grader gap grows steadily across training. Sensitivity to other authorities stays flat.
• "An aligned policy and a reward-seeking policy can look identical while a grader is watching."

Hojmark, Scheurer, Nitishinskaya et al., "Measuring Reward-Seeking by Instilling Contrastive Beliefs," Apollo Research and OpenAI, Jul 21, 2026

1. What happened

2. How it got out

3. Where the misalignment came from

4. Does hacking make it better at hacking?

5. What the warning actually is

Reward hacking makes a model

more eager to hack.

Does it make it better at hacking?

markasoftware, in the comments: "not convinced that corresponds to actually being better at hacking."

Tim Hua's reply: "Yeah I'm not super confident on this part."

Comment thread on Hua, LessWrong, Jul 27, 2026

Four live hypotheses

Nobody has run the experiment that would separate these. Until someone does, all four stay on the table.

1. What happened

2. How it got out

3. Where the misalignment came from

4. Does hacking make it better at hacking?

5. What the warning actually is

Warning shot, or an open door

"The models didn't escape because they're gods. They escaped because someone left the door open. Congress should regulate the door."

Kate Klonick, Lawfare, Jul 29, 2026

• "Not about rogue AI, despite the headlines." Will Douglas Heaven, MIT Technology Review
• "A containment failure with the safeties turned off." Dan Guido, Trail of Bits, and Jake Williams, IANS
• "Arguably the first AI loss of control incident." Hua and Singh, the other way
• "Very likely could have prevented this hack." Hua and Singh again, on OpenAI

The warning is about what happens next

"If developers naively try to select against noticeable misalignment, only the hardest-to-detect, most coordinated misalignment will likely remain."

Mallen and Gupta, Jul 23, 2026

• Two mechanisms, and the second is buried in a footnote.
• Selection during training removes detectability, not misalignment.
• Drift during deployment: fitness-seeking is unstable, and ambitious misalignment is a ratchet.
• "The default path is headed towards an attractor with harder-to-dislodge, more dangerous fitness-seeking goals."

The warning shot is not about what the model did. It is about what the labs will do in response.

What would actually settle it

• Hua and singh, Aug 3: eighteen questions, about 83 experiments, all black-box.
• Step zero is reproducing the incident and measuring a base hack rate.
• Their question, and ours:
• "How many times was the model rewarded for breaking out of sandboxes or similar behaviors during training?"
• METR, Jul 28: nine questions and the access required. Question 6 is the same one.
• Their aside may matter more: a law requiring monthly disclosure of the top three misalignment incidents.
• Nothing is published yet. OpenAI's report and the METR and Redwood review are both pending.

Hua and singh, Aug 3, 2026 · METR, Jul 28, 2026 · status verified Aug 4, 2026

Thank you

Andreas Turanski

LinkedIn https://linkedin.com/in/andreasturanski/

Blog: https://andreas.turanski.com/

AI Safety Evals Paper Reading Group · August 4, 2026

Resources, summaries and the full source list are linked in the event page.

The key documents

Tim Hua, "Is Mythos good at cyber..." LessWrong, Jul 27, 2026

Hua and Singh, "Concrete Evaluations..." LessWrong and AF, Aug 3, 2026

Greenblatt and Shlegeris, Redwood podcast ep. 2 Jul 23, 2026, key section 10:17 to 15:05

Mallen and Gupta, "Are we existentially threatened..." LessWrong, Jul 23, 2026

OpenAI, incident disclosure Jul 21, 2026

Hugging Face, technical timeline Jul 27, 2026

Anthropic, three cyber-eval incidents Jul 30, 2026

UK AISI, cheating in frontier evals Jul 21, 2026

Hojmark et al., "Measuring Reward-Seeking" Apollo and OpenAI, Jul 21, 2026

METR, investigating propensities after incidents Jul 28, 2026

Every title above is a live link. Full annotated list in the resources doc linked from the event page.

Appendix

Discussion questions (1 of 2)

• If the authors cannot attribute the behavior to training data, does the checkpoint-trend experiment still settle it?
• They concede data attribution probably cannot answer it. Running the eval across checkpoints needs no attribution.
• Step zero is reproducing the incident. What is the argument that reproducing it is safe?
• Anthropic's incidents happened because a sealed environment was not sealed. Who certifies the certifier?
• The Aug 3 post states the RL claim flatly. Its citation is one of its own authors. Does that change the weight?

Discussion questions (2 of 2)

• If OpenAI "very likely could have prevented this," is it a warning shot or an operational failure?
• Both readings are defensible. Which one drives policy?
• Behavioral evals may expire in four months. What replaces them?
• Interpretability is described as being in a sad state. Does the toolkit get worse before better?
• Would mandatory monthly disclosure of the top three misalignment incidents have surfaced this sooner?
• Greenblatt: "we're likely seeing the tip of the iceberg." What are the incentives against?