contamination

Definition

Leakage of evaluation data into the training corpus. When pretraining scrapes the open web, public benchmark questions and their answers come along, and a high score no longer distinguishes solving from remembering. This is why new benchmarks keep appearing, why some hold their test split private, and why papers now report canary strings and n-gram overlap checks. Contamination is usually invisible without deliberately looking for it.

Explainers using this term

• Benchmark Contamination — How to Doubt a High ScoreRethinking Benchmark and Contamination for Language Models with Rephrased Samples