benchmarking

Definition

The act of measuring and comparing systems on a shared task suite under matched conditions. A title containing "benchmarking" usually signals an evaluation paper rather than a new method — the contribution is the measurement, the protocol, or the finding that reported numbers do not hold up. Recurring concerns in this literature are contamination, where test items leak into training data, and whether the metric actually tracks the capability it claims to.

Explainers using this term

• Paper Explained: When Wording Changes an AI Reviewer's Score — RobustReview and SciCoreA Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review
• Paper Walkthrough: LoopArena — Benchmarking the Model That Steers a Coding AgentLoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
• Paper Walkthrough: SemComp-Bench — Asking Video Generators 'So Did It Actually Get Done?'SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation