benchmarking

Definition

The act of measuring and comparing systems on a shared task suite under matched conditions. A title containing "benchmarking" usually signals an evaluation paper rather than a new method — the contribution is the measurement, the protocol, or the finding that reported numbers do not hold up. Recurring concerns in this literature are contamination, where test items leak into training data, and whether the metric actually tracks the capability it claims to.

Explainers using this term

• Paper Walkthrough: LoopArena — Benchmarking the Model That Steers a Coding AgentLoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
• Paper Walkthrough: SemComp-Bench — Asking Video Generators 'So Did It Actually Get Done?'SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation
• Paper Walkthrough: MerchantBench — Can an LLM Agent Run an Online Store for a Year? Why It Earns Only 27.3% of What Humans DoMerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations