benchmark

Definition

A standardized package of tasks, data, and scoring rules that lets different methods be compared on the same footing. Benchmarks steer the field: whatever the leaderboard measures is what gets optimized, which is why contamination — test items leaking into pre-training data — and quiet overfitting to one suite are recurring concerns. A benchmark is more than a dataset; it fixes the evaluation protocol as well.

Explainers using this term

• Paper Explained: Last Translation Benchmark — Measuring Translation with Breaking Examples and Verification RulesLast Translation Benchmark
• Benchmark Contamination — How to Doubt a High ScoreRethinking Benchmark and Contamination for Language Models with Rephrased Samples