benchmarks

Definition

Fixed datasets paired with a fixed evaluation protocol, used so that competing methods can be compared under the same conditions. Benchmarks steer whole subfields, which is why flaws in them matter so much: if the task is too easy or the labels are unreliable, scores rise without capability rising. A dataset is raw material; a benchmark is the dataset plus the rules for scoring on it.