harness

Definition

The scaffolding around a model that actually runs an evaluation or a task: loading examples, formatting prompts, parsing outputs, scoring. It exists so comparisons are apples-to-apples. In practice harness details — prompt template, few-shot count, answer extraction, stop tokens — move benchmark numbers enough that scores from different harnesses are not comparable, a common source of contradictory reported results.

Explainers using this term

• Paper Walkthrough: HarnessDev — Can an LLM Build and Maintain the System It Runs Inside?HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
• Paper Walkthrough: Training Agents to Evolve with Their HarnessTraining Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report
• Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure LogsAutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces