evaluation

Definition

Measuring model quality against defined tasks and metrics. The credibility of an evaluation rests on unglamorous details: whether test data leaked into training, whether the metric tracks what you actually care about, and whether the gap between two systems exceeds run-to-run variance. Benchmark numbers alone rarely predict deployment usefulness, which is why human and task-grounded evaluations sit alongside them.

Explainers using this term

• Paper Explained: Can MiniMax-H3 Reason About the Physical World? Testing an Omni-Modal Generator on What the Prompt Leaves OutCan MiniMax-H3 Reason About the Physical World? An Evaluation of Omni-Modal Generative Model
• Paper Explained: DataFlex-RL — Did 11 RLVR Data Policies Beat Uniform Sampling? Re-measured Over 12 SeedsDataFlex-RL: An Evaluation Platform for RLVR Data Policies
• Paper walkthrough: RealSWE — re-measuring coding agents when the request is just "fix it"RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

In the textbook

• Before You Say It Got Better — Building an Evaluation