evaluators

Definition

Whoever assigns the scores — human annotators, or a model put in the judge seat. Automatic pipelines lean on model evaluators because they are cheap and reproducible, but the evaluator's own quirks leak into the numbers, so measuring and correcting judge bias is now its own line of work. Distinguish it from the benchmark, which is the data being scored, not the thing doing the scoring.