LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem

A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge.

Measuring Massive Multitask Language Understanding

What "top score on the benchmark" is actually claiming

Every new model arrives with a row of bar charts. MMLU, GSM8K, HumanEval — different names, same posture: higher is smarter. But to read one of those bars you need to know at least three things. What was measured, how it was graded, and whether the model had already seen the questions.

The closest everyday analogue is a practice exam. Scoring in the 99th percentile means 99th percentile on this exam, against this pool of test-takers, for this style of question — not a guarantee about the real thing. And if one of the test-takers got hold of the question booklet a week early and memorised it, that percentile measures recall, not ability. Both failure modes show up in LLM evaluation: the target drifts away from what you care about, and the questions leak.

Opening up MMLU

MMLU stands for Massive Multitask Language Understanding, a multiple-choice test released in 2020 (Hendrycks et al.). The construction is plain: 57 subjects, four options each, more than ten thousand questions, assembled from human exams spanning high-school physics through law and medicine.

Because there are four options, answering at random already scores 25. The floor is not zero, so a score has to be read as growth above 25 — otherwise you will systematically overstate how much any improvement bought you.

The bigger lever, and the one that never appears on the chart, is the grading method. There are two common ways to make a model answer a four-way question.

1. Generate and extract: let the model write "The answer is B" and pull the B out of the text.
2. Score by likelihood: compute the probability the model assigns to each of A/B/C/D as a continuation, and take the highest.

The same model on the same questions scores differently under these two. The first depends on how good your answer-extraction code is; the second never observes the model speaking freely at all. So when comparing MMLU numbers across papers and model cards, the first thing to check is whether the scoring method and the few-shot count (how many worked examples were shown) match. Putting mismatched numbers on one axis is like plotting quantities with different units.

It is worth being equally explicit about what the format cannot see. Four-way multiple choice measures the ability to pick from a supplied list. Whether the model can justify the pick, whether it can say "I don't know" when it doesn't, whether it holds a long context, whether it follows fine-grained instructions — most of what determines whether a product feels good lives outside this format. That gap is where "great benchmarks, disappointing in practice" comes from.

A score is an interval, not a number

A benchmark score is a sample mean: you drew $n$ questions from the space of possible questions and measured on those. Samples carry error.

Here $p$ is the accuracy (between 0 and 1), $n$ is the number of questions, and $\mathrm{SE}$ is how much the score would bounce around if you re-measured an equally capable model. The bounce shrinks with the square root of the question count.

Put numbers in. At $p=0.85$ and $n=14{,}000$, $\mathrm{SE}$ is about 0.3 points, so a 95% interval is roughly ±0.6 points. At $n=200$ — and plenty of hard benchmarks are that size — it is about 2.5 points, an interval of ±5. Which means "we won by three points" on a 200-question benchmark says essentially nothing. And this is only the error from which questions got picked; sampling randomness, the choice of few-shot examples, and the exact wording of the prompt all stack on top of it.

Relabel the figure: "training data" is the public benchmark, "test data" is what your actual users type. That relabelling describes the whole industry. Everyone reads MMLU when selecting training data and tuning prompts. Each individual decision is legitimate, but once the same held-out set has been consulted a few hundred times, it has effectively become training data. This is the same mechanism as overfitting to a validation split, covered in Overfitting and Evaluation Design, running at the scale of an entire field. Which leads directly to the harder version of the problem: the questions themselves ending up inside the models.