standardized

Definition

Made uniform in format, procedure or conditions. In benchmarking this is a substantive contribution: unless environment, task set and scoring are held fixed, reported numbers are not comparable across papers, and much apparent progress dissolves under closer reading. The cost is that a frozen standard also freezes what gets measured, so standardisation and coverage pull against each other.

Explainers using this term

• Paper Explained: OSReward — Can You Trust the AI That Grades AI? Remeasuring Rewards for Computer-Use AgentsOSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models