LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem

A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge.

Measuring Massive Multitask Language Understanding

What "top score on the benchmark" is actually claiming

Every new model arrives with a row of bar charts. MMLU, GSM8K, HumanEval — different names, same posture: higher is smarter. But to read one of those bars you need to know at least three things. What was measured, how it was graded, and whether the model had already seen the questions.

The closest everyday analogue is a practice exam. Scoring in the 99th percentile means 99th percentile on this exam, against this pool of test-takers, for this style of question — not a guarantee about the real thing. And if one of the test-takers got hold of the question booklet a week early and memorised it, that percentile measures recall, not ability. Both failure modes show up in LLM evaluation: the target drifts away from what you care about, and the questions leak.

Opening up MMLU

MMLU stands for Massive Multitask Language Understanding, a multiple-choice test released in 2020 (Hendrycks et al.). The construction is plain: 57 subjects, four options each, more than ten thousand questions, assembled from human exams spanning high-school physics through law and medicine.

Because there are four options, answering at random already scores 25. The floor is not zero, so a score has to be read as growth above 25 — otherwise you will systematically overstate how much any improvement bought you.

The bigger lever, and the one that never appears on the chart, is the grading method. There are two common ways to make a model answer a four-way question.

1. Generate and extract: let the model write "The answer is B" and pull the B out of the text.
2. Score by likelihood: compute the probability the model assigns to each of A/B/C/D as a continuation, and take the highest.

The same model on the same questions scores differently under these two. The first depends on how good your answer-extraction code is; the second never observes the model speaking freely at all. So when comparing MMLU numbers across papers and model cards, the first thing to check is whether the scoring method and the few-shot count (how many worked examples were shown) match. Putting mismatched numbers on one axis is like plotting quantities with different units.

It is worth being equally explicit about what the format cannot see. Four-way multiple choice measures the ability to pick from a supplied list. Whether the model can justify the pick, whether it can say "I don't know" when it doesn't, whether it holds a long context, whether it follows fine-grained instructions — most of what determines whether a product feels good lives outside this format. That gap is where "great benchmarks, disappointing in practice" comes from.

A score is an interval, not a number

A benchmark score is a sample mean: you drew $n$ questions from the space of possible questions and measured on those. Samples carry error.

In words, SE is the typical amount the score would bounce around if you re-measured an equally capable model on a fresh set of questions. It shrinks only with the square root of the question count, so quadrupling $n$ halves the error.

Here $p$ is the accuracy (between 0 and 1), $n$ is the number of questions, and $\mathrm{SE}$ is how much the score would bounce around if you re-measured an equally capable model. The bounce shrinks with the square root of the question count.

Put numbers in. At $p=0.85$ and $n=14{,}000$, $\mathrm{SE}$ is about 0.3 points, so a 95% interval is roughly ±0.6 points. At $n=200$ — and plenty of hard benchmarks are that size — it is about 2.5 points, an interval of ±5. Which means "we won by three points" on a 200-question benchmark says essentially nothing. And this is only the error from which questions got picked; sampling randomness, the choice of few-shot examples, and the exact wording of the prompt all stack on top of it.

Relabel the figure: "training data" is the public benchmark, "test data" is what your actual users type. That relabelling describes the whole industry. Everyone reads MMLU when selecting training data and tuning prompts. Each individual decision is legitimate, but once the same held-out set has been consulted a few hundred times, it has effectively become training data. This is the same mechanism as overfitting to a validation split, covered in Overfitting and Evaluation Design, running at the scale of an entire field. Which leads directly to the harder version of the problem: the questions themselves ending up inside the models.

Contamination — the questions end up in the training data

Pretraining data comes from large web crawls. Public benchmarks live on GitHub, in explainer blog posts, and on dataset-sharing sites. The more famous a benchmark gets, the more copies exist, and the higher the odds it lands in the next model's training set. Contamination is not someone being careless; it is the structural consequence of publishing a benchmark. The symptoms are the symptoms of memorisation: rewording the question slightly breaks it, shuffling the answer options costs points, the final answer is right while the working that led there is incoherent.

GSM1k measured this head-on (Zhang et al., 2024). The authors built 1,250 brand-new problems matching the difficulty distribution and format of GSM8K, the grade-school arithmetic benchmark, and scored models on both. The result was not uniform: some model families barely moved, while others dropped by double-digit points. The same "GSM8K score" had been referring to different things depending on the model.

Three countermeasures are in common use. N-gram overlap drops any evaluation item that shares a contiguous span above some threshold (13 words, 50 characters) with the training corpus; it catches blatant copies and misses paraphrases. Canary strings embed a globally unique identifier in the benchmark files that exists nowhere else on the web — if a model can reproduce the continuation, it saw the file (BIG-bench uses this). Cutting by time scores models only on items created after their training cutoff, which is what rolling benchmarks like LiveBench and LiveCodeBench operationalise by continuously swapping in fresh questions. It is the soundest of the three, and the hardest to run fairly, because different models have different cutoffs.

You cannot prove the absence of contamination. If the training data is undisclosed, detection is impossible in principle, and paraphrased copies never trip a string match. What you can do is accumulate evidence: re-measure on fresh questions that contamination would have degraded, and show that the score held.

Automated grading — correct answers marked wrong

Grading ultimately runs as code, and that code moves scores far more than people expect. The classic case is answer extraction.

import re

def extract_choice(text):
    m = re.search(r"\b([A-D])\b", text)   # take the first A-D that appears
    return m.group(1) if m else None

extract_choice("A confuses voltage with current, so the answer is C.")  # -> 'A'

The model answered C correctly; the grader recorded A and marked it wrong. Instruct the model to emit only the letter and extraction becomes reliable — but now instruction-following ability is mixed into what was supposed to be a subject-knowledge score. Either way, the thing being measured has shifted.

For code generation the standard metric is pass@k: generate $k$ samples, count the item as solved if any of them passes the tests. Generating exactly $k$ makes the estimate noisy, so the usual practice is to generate $n$ samples ($n > k$) and compute an unbiased estimate.

In words, pass@$k$ is the chance that at least one of $k$ samples passes the tests. It is computed as 1 minus the probability that all $k$ samples drawn from your $n$ generations are failures.

$n$ is the number of samples drawn per problem, $c$ is how many of them passed the tests, and $\binom{a}{b}$ is the number of ways to choose $b$ items from $a$. The fraction is the probability that drawing $k$ samples yields nothing but failures, so subtracting it from 1 gives the probability of at least one success.

pass@1 and pass@10 describe different abilities. A high pass@10 means the model can produce a candidate pool that contains a winner — and if your product has no mechanism to run the tests and pick, that number never reaches the user.

Three biases when an LLM does the grading

Free-form answers cannot be string-matched, so the common workaround is to have a strong model grade them: LLM-as-a-judge. The MT-Bench / Chatbot Arena paper (Zheng et al., 2023) showed that this agrees with human judgement reasonably well, and that it carries distinct biases.

• Position bias: present two answers as A and B and the presentation order itself shifts who wins.
• Verbosity bias: longer answers are preferred. Adding words raises the win rate even when no information was added.
• Self-enhancement bias: a judge rates its own outputs — or those of related models — more highly.

The fix for position bias is crude and effective.

def judge_pair(a, b):
    v1 = ask_judge(a, b)          # grade in A-then-B order
    v2 = ask_judge(b, a)          # swap the order and grade again
    if v1 == v2:
        return v1                 # only count it when both orderings agree
    return "tie"                  # disagreement = position bias suspected

You pay double the calls, and order effects mostly disappear. For verbosity, state in the rubric that length is not a criterion, then log answer length separately and check the correlation afterwards. For self-enhancement, the reliable move is using a judge from a different model family than anything being judged. Grading your own model improvements with your own flagship model may fly internally, but it is not a number you can publish.

Chatbot Arena — turning votes into a single number

In Chatbot Arena (now LMArena), you submit a question, two anonymous models answer side by side, and you vote for the better one. Names are hidden, so brand pull is dampened; questions come from real users, so the item set is never fixed; and because the questions did not exist beforehand, contamination is much harder.

What comes out is a pile of match records. The Bradley-Terry model turns that pile into one rating per model by assigning each model a strength $\xi_i$ and writing the win probability as:

In words, the chance that model $i$ beats model $j$ depends only on the gap between their strengths, not on their absolute values. A bigger gap pushes the win probability further toward 1 for the stronger model.

$\xi_i$ is the strength of model $i$ and $\sigma$ is the sigmoid function. What the equation says is that the win rate depends only on the difference in strength, never on the absolute values. All the $\xi$ values are estimated jointly so as to make the observed wins and losses as likely as possible (maximum likelihood). It is the same shape as chess Elo — Elo's familiar "a 400-point gap means roughly a 91% win rate for the stronger side" is this formula with the scale rewritten.

The weaknesses are visible from the same equation. First, preference is not correctness. Voters pick the answer that felt better, and style — tidy bullet points, an agreeable length — is known to attract votes on its own, which is why the operators publish a style-controlled ranking alongside the raw one. Reading the uncontrolled leaderboard as a capability ranking is a mistake. Second, the question distribution is the voter distribution: what people bring to an arena site is not your domain, whether that is clinical notes, contracts, or your internal codebase. Third, the top entries are often separated by a few dozen rating points, and without enough matches their confidence intervals overlap. First and third place being statistically indistinguishable is routine.

How this plays out on the job

The people running evaluations are ML engineers, dedicated eval owners, and whoever decides which model ships in the product. In practice you start with an off-the-shelf harness: EleutherAI's lm-evaluation-harness lets you pin conditions explicitly with flags like --tasks mmlu and --num_fewshot 5. Note that its acc and acc_norm (accuracy normalised by option length) are different quantities — never mix them in one comparison. Stanford's HELM is built to report robustness, bias, and efficiency next to accuracy, and is worth reading as an implementation of the "don't rank on one number" principle. If all you need is a regression test at the prompt level, something light like promptfoo is enough.

Building an in-house evaluation set follows a fixed recipe. Draw the items from production logs — public benchmarks are for first-pass screening only. Split a dev set you look at on every iteration from a test set you touch only for the final call (the moment you look at test repeatedly, it has become dev). Run at temperature 0 with a fixed seed, at least twice. Compute $\sqrt{p(1-p)/n}$ and report the result with its width: "82% ± 3pt".

Three pitfalls turn into incidents:

• Letting a public benchmark into your training data. Distributed instruction-tuning datasets sometimes contain items from the very benchmarks you plan to report, so run an n-gram overlap filter before using them.
• Confusing "percent" with "points". 80% → 84% is a four-point improvement, not a 4% one (that would be 83.2%).
• Using a special prompt only at evaluation time. Few-shot examples and system prompts tuned for the eval harness drift away from production. How much the phrasing alone moves results is covered in The Science of Prompt Engineering.

Three questions come up in design review: how did you check for contamination, is that difference significant, and does the metric match the use case. None of them asks for a high score; all three ask how the number was made. And it is worth noticing that the Arena's structure — estimating strength from pairwise match records — is nearly identical to the reward model in Instruction Tuning and RLHF. Evaluation and training turn out to be two sides of the same equation.

Takeaways

• A score only means something when it comes with how it was measured (scoring method, few-shot count) and the error bar implied by the question count.
• Contamination is a structural consequence of publishing benchmarks. You cannot prove it is absent; you can only show that fresh questions did not knock the score down.
• LLM-as-a-judge carries position, verbosity, and self-enhancement biases. Double-grading with the order swapped removes most of the position effect.
• Chatbot Arena resists contamination but measures preference. Do not read the style-uncontrolled ranking as a capability ranking.
• The final call belongs to an evaluation set from your own domain, not to a public benchmark.