Paper Explained: StarHarness — Evolving the Scaffold Instead of the Weights

Freeze the model weights and search over the harness instead — prompts, tool schemas, skills, MCP providers, subagents, agent loop. Across three enterprise benchmarks this bought 20–35 points, held on tasks never used for evolution, and transferred to other models unchanged.

StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments

A great cook in an unfamiliar kitchen

Drop an excellent cook into a kitchen they have never worked in. They do not know where the knives live, they have not learned the quirks of the burners, and in this particular kitchen the jar labelled "salt" is full of sugar. The cook's skill — the model weights, in our analogy — has not changed at all. The food that comes out is still terrible.

This is roughly what happens when you put an LLM agent on top of a real enterprise system. The model is smart enough. But the internal API schema is stricter than the tool description admits, and nobody wrote down that changing an incident's priority means you also have to update impact and urgency. The paper calls this persistent model–environment mismatch.

The paper is "StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments" (ServiceNow / Mila / Université de Montréal, arXiv:2608.24804, published 2026-08-25).

Here is the abstract in plain terms. StarHarness is a framework for evolving an environment-specific agent harness while the model weights stay fixed. What gets evolved includes prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration. It builds a compact evolution pool by stratifying tasks according to how the baseline fails on them, separates proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for measuring generalization. Across ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, harness evolution improved full-benchmark performance by 20–35 percentage points over the default harness after only 4–12 accepted changes per environment. The gains persisted on tasks excluded from evolution and transferred across GPT and Qwen model families without re-evolution. Trace analysis links the improvement to interface repairs, environment conventions, and operational knowledge that compresses search — with fewer false-positive diagnoses and shorter trajectories in several settings.

What exactly is a "harness"?

A harness is the tack you strap onto a horse — not the horse. For agents, it means all the executable scaffolding surrounding the model. The paper's implementation is specific about the editable surface (§3.1): prompt and task framing, tool definitions and schemas, argument preprocessing, skills, MCP providers, subagent structure, context management, verification, and finish logic.

The important consequence: all of this is just code. None of it lives inside the weights, so a change can be proposed as a git diff, tested, and reverted. The paper draws this contrast with weight updates explicitly (§2.1).

The setup is two-layered. The optimizing side is a coding harness built on Oh My Pi, a variant of the Pi agent harness; it hosts the proposer and runs the edit–validate–evaluate loop. The optimized side is a separate harness called Stirrup, which is the agent that actually solves the benchmark. Model weights and benchmark stay fixed throughout evolution.

What is being maximized

The objective fits on one line (§3.1).

Reading it out loud: $h$ is a harness (the code listed above), $\mathcal{H}$ is the space of permitted harnesses, and $J(h;\mathcal{D})$ is the mean task score you get by running the fixed model $M$ with harness $h$ over a task set $\mathcal{D}$, higher being better. So equation (1) says only this: pick the harness that maximizes the mean score on a task set that was never used during evolution.

And of course you cannot peek at held-out outcomes while searching. So StarHarness approximates that target with proposer-visible search tasks and proposer-hidden selection tasks. That approximation is the heart of the method.

Why "stratified"? Building the evolution pool

Running the whole benchmark on every iteration is expensive. Picking half of it arbitrarily is worse: you end up with a harness tuned to that particular half. That is plain overfitting.

Before evolution starts, StarHarness keeps the $N'$ tasks (out of $N$) whose evaluation reproduces, then samples an evolution pool of $K \approx N/2$ tasks (§3.2). The sampling uses three descriptors computed from a baseline run:

• Baseline failure mode (e.g. wrong_tool, context_loss, missing_evidence, premature_conclusion)
• Baseline task score
• Verifier pass rate

The pool is then split into search tasks the proposer may inspect and selection tasks hidden from it, matching the two splits on score, failure-mode, and verifier-pass distributions. The proposer gets search-task traces and outcomes, but never selection-task contents, traces, verifier feedback, or per-task outcomes. The remaining $N'-K$ tasks are the holdout, and they influence neither proposal nor acceptance.

How the search runs: hill climbing and tree search

The same proposer, validator, evaluator, and acceptance score are used in two search procedures (§3.3).

In hill climbing, the state is a single frontier harness. Each iteration, the proposer reads traces of the current frontier and emits one patch $\Delta_t$. It is kept only if the selection score strictly improves, or ties while improving an available verifier metric. Otherwise the frontier is restored.

In tree search, the state is a set of candidate nodes, each storing a parent pointer, cumulative patch, search traces, validation status, and selection score. The proposer may explore a failure pattern, draft a patch, debug a failed candidate, merge two compatible nodes, or improve an existing one. Valid nodes are scored on the same hidden selection set, and the best survivor becomes the next frontier. The point is to keep alternative hypotheses alive rather than committing to the first accepted edit.

The paper is careful about what this does and does not show: the two modes were used as an exploration–exploitation control experiment on EnterpriseOps-Gym alone, and because the stages are sequential it describes how the modes complement each other rather than providing a causal head-to-head comparison (§3.3).

Stopping the proposer from cheating

The proposer can read benchmark traces, so left alone it would start writing "if task ID is 17, answer X". Guardrails forbid exactly that (§3.4):

• Branching on task IDs, or hard-coded answers
• Verifier or assertion content in agent prompts
• Ground-truth tables or hidden-state access
• Benchmark-specific answer mappings

Every candidate is a git diff against the current frontier, scoped to the benchmark's editable directories and the shared agent framework. The validator checks scope, imports, and a single-task smoke test; anything that fails is reverted before the expensive selection evaluation runs.

There is one more cost gate ahead of selection evaluation: a test flip (§3.1). The proposer nominates a single task, and if the candidate fails to flip that task, the loop records a rejection and skips evaluation entirely. Accepted, rejected, invalid, and crashed candidates all land in a persistent memory ledger that carries frontier scores, per-task outcomes, accepted hypotheses, and discarded attempts into later iterations.