Predictable Noise in LLM Benchmarks [10/27]

JavaScript isn't enabled in your browser, so this file can't be opened. Enable and reload.

Email *

Based on the motivation presented, why should we be cautious about a paper claiming a 2.5% improvement on the HumanEval benchmark?

1 point

(A) The HumanEval benchmark is outdated and no longer relevant.

(B) The test set for HumanEval is very small (N=164) , meaning a 2.5% gain is likely not statistically significant and falls within the benchmark's high noise level.

(D) The HumanEval benchmark only tests Python, not other languages.

The idea that a few "hard" generative problems might be more informative than many easy ones. What did the exploratory data analysis (the heatmaps) reveal about this? *

1 point

(A) It revealed significant inconsistency, where strong models still fail easy problems and weaker models sometimes pass hard ones , undermining the reliability of small, hard test sets.

(B) It confirmed this hypothesis, showing that better models consistently solve harder problems.

(D) It proved that model performance is based entirely on memorization.

What was the key finding from analyzing millions of prompt results regarding the statistical noise of a benchmark in the lecture? *

1 point

(A) Noise is completely random, making it impossible to analyze.

(B) Noise is predictable and follows a "characteristic noise curve" that depends on the model's overall accuracy , which we found can be modeled by a Beta distribution.

(D) Noise can be completely eliminated by using a paired statistical test.

The speaker introduced the "signal-to-noise ratio" to measure a benchmark's quality. What was the main conclusion from this analysis? *

1 point

(A) Complex generative benchmarks like HumanEval and MBPP+ have a very high signal-to-noise ratio and are the most reliable.

(B) The signal-to-noise ratio is poor for all benchmarks, so we should stop using them.

(C) Simple, large multiple-choice benchmarks (like MMLU and gsm8k) currently have a much better signal-to-noise ratio than most small, complex code benchmarks (like HumanEval).

(D) Doubling a model's size always results in a reliable improvement across all benchmarks.

The attempt to "boost the signal" by reweighting or filtering for harder problems failed. What was the main reason for this failure? *

1 point

(A) Subjectively "high quality" datasets like HumanEval+ already had a perfect signal-to-noise ratio.

(B) Item Response Theory (IRT) proved that all problems have equal difficulty.

(D) The models are the bigger source of inconsistency than the benchmarks themselves , with even bad models sometimes solving hard problems (perhaps via memorization).

A copy of your responses will be emailed to the address you provided.

Submit

Clear form

reCAPTCHA

Privacy Terms

This form was created inside of UC Berkeley.

Does this form look suspicious? Report

Forms