Large Language Monkeys: Scaling Inference-Time Compute with Repeated Sampling

Brad Brown*, Jordan Juravsky*, Ryan Ehrlich*, Ronald Clark, Quoc Le, Chris Ré, Azalia Mirhoseini

​

​

​

Why Care About Scaling Inference-Time Compute?

1. Unexploited Axis for Scaling

• Training = $billions, inference OOMs less

​

​

2. Concentrated Axis for Scaling

- Spin ‘til you win on the problem you care about.

Why Care About Scaling Inference-Time Compute?

1. Unexploited Axis for Scaling

• Training = $billions, inference OOMs less

​

​

2. Concentrated Axis for Scaling

- Spin ‘til you win on the problem you care about.

We Did the Simplest Possible Thing

How far can we get with a for loop?

Main Findings:

Repeated sampling:

​

1. Is effective across multiple tasks and models.

​

• Can often yield predictable benefits (i.e. scaling laws).

​

• Requires scalable verification.

Main Findings:

Repeated sampling:

​

1. Is effective across multiple tasks and models.

​

• Can often yield predictable benefits (i.e. scaling laws).

​

• Requires scalable verification.

Main Findings:

Repeated sampling:

​

1. Is effective across multiple tasks and models.

​

• Can often yield predictable benefits (i.e. scaling laws).

​

• Requires scalable verification.

Main Findings:

Repeated sampling:

​

1. Is effective across multiple tasks and models.

​

• Can often yield predictable benefits (i.e. scaling laws).

​

• Requires scalable verification.

How Do We Evaluate Repeated Sampling?

Coverage: As the number of samples increases, what fraction of problems can we solve using any sample that was generated?

​

Repeated Sampling is Effective Across Tasks

Repeated Sampling is Effective Across Tasks

pass@250 = 56% on SWE-bench Lite using DeepSeek-Coder-V2-Instruct

​

current single-attempt SOTA is 43%

Repeated Sampling can be Cost-Effective

Repeated Sampling is Effective Across Models

Main Findings:

Repeated sampling:

​

1. Is effective across multiple tasks and models.

​

• Can often yield predictable benefits (i.e. scaling laws).

​

• Requires scalable verification to exploit in practice.

Scaling Laws For Repeated Sampling

Scaling Laws For Repeated Sampling

Main Findings:

Repeated sampling:

​

1. Is effective across multiple tasks and models.

​

• Can often yield predictable benefits (i.e. scaling laws).

​

• Requires scalable verification.

Verification Difficulty Varies by Task

• In some settings, like code, formal proofs, etc., we have access to tools for automatically verifying solutions.

​

• In other settings, not so much. We’re forced to resort to methods like:
• Majority voting.
• Reward models.

Common Verification Methods Don’t Scale

Verification Requires Finding Needles in the Haystack

Future/Ongoing Work

• (In Progress) Building AI coders from the ground up that scale inference-time compute.
• Amortizing expensive ops (e.g. reading every file) across many samples.
• Let models write their own tests – another place to scale compute!

​

• Better algorithms than repeated sampling.
• Combining sequential and parallel scaling.
• Removing isolation between parallel branches.

​

​

• Designing scalable verification.

​

Thanks!

Training Scaling Laws Work Well!

Repeated Sampling can be Cost-Effective