harness

Definition

The scaffolding around a model that actually runs an evaluation or a task: loading examples, formatting prompts, parsing outputs, scoring. It exists so comparisons are apples-to-apples. In practice harness details — prompt template, few-shot count, answer extraction, stop tokens — move benchmark numbers enough that scores from different harnesses are not comparable, a common source of contradictory reported results.

Explainers using this term

• Paper Explained: Retrieval-Augmented Skill Optimization via Cross-Harness Adaptation — Borrow Someone Else's Skill, Rewrite It for Your Own HarnessRetrieval-Augmented Skill Optimization via Cross-Harness Adaptation
• Paper Walkthrough: WHALE — Training the Weights and the Harness in TurnsWHALE: A Simple Recipe for Joint Harness-Weight Optimization
• HarnessDev explained (arXiv:2609.01437) — agent harness benchmarkHarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?