Mitigating Goal Misgeneralization via Minimax Regret

Reinforcement Learning Conference 2025

Karim Abdel Sadek

UC Berkeley

Coauthors:

Matthew Farrugia-Roberts

Usman

Anwar

Hannah

Erlebach

Christian

S. De Witt

David

Krueger

Michael

Dennis

University of Oxford

University of Cambridge

University of Oxford

University of Oxford

MILA

Google DeepMind

Desiderata: AI systems that do what we want them to do

Some failure modes:

​

• The reward function we designed may not induced the desired behaviour -> reward hacking

​

reward hacking

​

• Give it a correct reward function to optimize for!

• policy fails out-of-distribution -> capability generalization

​

How to incentivize an RL agent to complete a task we care about?

Even if the reward specification is correct and the policy is capable, we may not get the behaviour desired

Training

…

Deployment

Which goal will the agent pursue?

Goal Misgeneralization:

A policy which was optimal during training competently pursues an unintended goal at test time

​

Setting: Proxy distinguishing distribution shift

Proxy Goal

True goal

Non-distinguishing

Distinguishing

First, in training…

For a small

Then, at test time…

For a large

Standard RL training induces policies that optimise for the maximum expected value (MEV) on the training distribution

MEV Policy:

MEV Objective:

Level distribution

Return on a single level

Maximum expected value(MEV) possibly suffers from goal misgeneralization

Theorem 1 (informal): training via the MEV objective permits goal misgeneralization under a proxy-distinguishing distribution shift if the proportion of distinguishing levels in training is too small.

Intuition: A policy that pursues the proxy goal in all levels achieves enough return on non-distinguishing levels to be approximately optimal.

Instead, we will show that a policy induced by the MiniMax Expected Regret (MMER) objective produces more robust and aligned policies

MMER Policy:

Expected Regret:

Main Idea: Use Minimax Expected Regret instead of Maximum Expected Value as as the training objective

MiniMax Expected Regret(MMER) provably mitigates goal misgeneralization

Theorem 2 (informal): MMER does not permit goal misgeneralization under any distribution shift within the adversary’s strategy space (at the equilibrium of the two-player zero sum game)

Intuition: Distinguishing levels have high regret, leading the adversary to play those until the agent’s policy pursues the true goal.

Empirically, MMER-based methods outperform MEV ones and mitigate goal misgeneralization

DR

PLR

ACCEL

1e-5

3e-3

1e-1

α

0

1

0.5

0

Return

Empirically, MMER-based methods outperform MEV ones and mitigate goal misgeneralization

Proportion of distinguishing levels in underlying training distribution (log)

Average Return Distg

Concluding:

Distinguishing Level

Domain Randomization (MEV)

ACCEL (MMER)

• A correct reward function and and a capable policy may not be enough -> goal misgeneralization

See the paper for more experiments, environments and theoretical results!

• Maximum Expected Value policies can suffer goal misgeneralization

• Minimax Expected Regret policies are robust to this failure mode, theoretically and in practice

• Future work:
• Better regret estimates
• Case where α=0