#contamination
4 articles
01
·Evaluation & Judging·★ MEMBER·PAPER·10 min read
Benchmark Contamination — How to Doubt a High Score
How to tell whether a benchmark score reflects ability or memorization. Covers the three kinds of contamination, the three detection tools — n-gram overlap, embedding neighbors, membership inference — and where each stops working, plus canary strings and time-based splits, ending in a checklist for reading papers skeptically.
02
·Evaluation & Judging·FREE·PAPER·11 min read
A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure
What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system.
03
·Large Language Models·★ MEMBER·PAPER·11 min read
LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem
A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge.
04
·Agents·★ MEMBER·PAPER·11 min read
Evaluating Agents — How Benchmarks and Harnesses Are Built
An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe.