harnesses

Definition

The scaffolding that actually runs an evaluation: loading tasks, formatting prompts, calling the model, parsing answers out of free text, retrying failures, and scoring. If a benchmark defines what is measured, the harness defines how — and its unglamorous details, such as few-shot format, stop tokens, and how a malformed answer is treated, routinely move reported scores by several points. Reproducing a number generally means reproducing the harness, not just the dataset.

Explainers using this term

• Paper Explained: StarHarness — Evolving the Scaffold Instead of the WeightsStarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments