Paper walkthrough: StateM — 95.3% on Terminal-Bench 2.1 and a USD 15 run, without touching a single weight
Long-horizon agents fail even when the model underneath can solve every individual step. StateM leaves the weights alone and hardens only the execution system around the agent, reporting 95.3% on Terminal-Bench 2.1 and a final scoring run billed at roughly USD 15 instead of USD 574.68. A ground-up walkthrough of the harness-scaling bet.
StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling
Primary source — what this article is built on
undefined2026-08-15→undefined2026-08-22same month
StateM: Reaching 95.3% Raw AccuracyZiheng Qin, Yaxin Lu, Zhangyang Atlas Wang et al. · 2026-08-15 · v1"arXiv:2608.15089Paper page·PDFor a $15 Frontier Runor a $15 Frontier Run
https://arxiv.org/abs/2608.15089"on Terminal-Bench 2.1 via Harness Scaling
undefined
Long-horizon agents can fail even when their underlying models can solve the constituent steps. They may lose track of mutable state, fail to reactivate lessons from earlier executions, skip known procedures, or stop prematurely. We bet on harness scaling to improve the execution system around an agent without changing its model weights. We introduce StateM, an agent-native runtime that organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh to 92.1\%, versus 83.1\% reference and GPT-5.6 Sol Ultra at 91.9\%. The runbook transfers unchanged to GPT-5.6. With GPT-5.6 Sol xhigh, StateM reaches 95.3\% raw accuracy across 445 trials and succeeds on all 89 tasks at least once. The frozen profile raises GPT-5.6 Luna from 76.7 to 85.4\%, above the 84.9\% Sol xhigh reference. Using the same runtime, runbook structure, and golden rules, less than \$38 of adaptation raises DeepSeek-V4 Flash from 82.7 to 88.1\% under standard timeouts and to 89.1\% on an 88-task common core. Extending only the remaining latency-sensitive task matches the reported 88.8\% GPT-5.6 Sol max result. Final-score API usage is about \$15 versus \$574.68 for the GPT reference; total DeepSeek expenditure is \$52.22. On BusinessBench, family-specific runbooks built on development sets yield held-out gains of 0.55 macro and 1.34 micro points; two mechanism-matched families improve by 10.04 points. Concrete rules generalize when tasks share execution structure, while the control methodology applies broadly. StateM turns selected postmortem findings into persistent, executable preconditions and practices, making learned controls explicit and enforceable through stateful controls. Code at github.com/henryqin1997/statem.
Every step is solvable. The whole run still falls over.
Picture a three-hour dinner menu. Chop the onions, sweat them on low heat, simmer for thirty minutes — pull any single step out and anyone who has cooked before can do it. Cook the whole thing end to end for the first time, though, and it usually goes wrong. You lose track of whether the salt went in. You don't recall the lesson from the batch you burned last month. You skip one step of the house method. And you turn the heat off early because it looks close enough.
That is exactly where this paper starts. Long-horizon agents can fail even when their underlying models can solve the constituent steps (Abstract). The paper names four failure shapes:
- losing track of mutable state
- failing to reactivate lessons from earlier executions
- skipping known procedures
- stopping prematurely
Line them up and none of them is a knowledge problem. Every one is a case of knowing the thing and mismanaging the execution — not the cook's skill, the kitchen's choreography.
That distinction is the fork in the road. If the cause is missing knowledge, the only lever is a better model: more training, more RL, more parameters. If the cause is choreography, there is a real chance of fixing it without touching the model at all.
Harness scaling: sharpening the vessel, not the weights
The paper states its bet plainly: "We bet on harness scaling to improve the execution system around an agent without changing its model weights" (Abstract).
The harness is the machinery you run the model inside. On any single agent turn, the model decides one thing — what to do next — and everything around that decision is code a human wrote. How many times the loop spins. How a tool result gets stringified back into context. When to give up. Where to resume from after a failure. All of that outer shell is the harness.
The word "scaling" is doing real work here. Recent scaling stories have been about parameters, data, and inference-time thinking. This paper adds a fourth axis: scale up how carefully the vessel is built. And unlike the other three, improvements to the vessel should in principle survive a model swap. Fine-tuning starts over every time a new model lands; choreography rules might carry across. Whether they actually carry is precisely what the paper measures.
The math of why long jobs fall over
Here is the crudest possible estimate of what more steps does to you.
is the per-step success rate, is the number of steps, and the left side is the probability of getting all the way through. Spelled out in words, it reads: multiply your per-move success rate by itself, once per move. Ten moves, ten multiplications — no other machinery. It assumes each step succeeds or fails independently.
What makes equation (1) vicious is that a high doesn't save you. An expert at with gets — two runs out of three die somewhere in the middle. Push from 0.99 to 0.995 (halving the error rate) and you recover to . In other words, on long tasks a small difference in per-step reliability becomes an order-of-magnitude difference end to end.
Equation (1) is not from the paper; it is a generic estimate this article supplies to make the difficulty tangible. Real steps are not independent, and real runs can recover mid-flight. The shape holds anyway: the more moves there are, the more exponentially the harness's reliability matters.
So if the vessel can enforce "don't drop state," "don't skip steps," and "don't stop early," the end-to-end rate should move without the model changing at all. That is the paper's setup.
What StateM is made of
StateM is described as an agent-native runtime that organizes execution around five things (Abstract).
| Component | What it does (per the abstract) |
|---|---|
| Durable states | Execution state is held as something that does not evaporate |
| Phase-local context | Only the context the current phase needs is put in front of the model |
| Checked transitions | Moving to the next phase requires passing a check |
| Recoverable runbooks | Work follows a procedure, and can be recovered when it breaks |
| Versioned procedural practices | Learned practice accumulates with versions, inspectable by agents and users together |
Set that against the four failure shapes and the design intent shows through. Losing state → durable states. Skipping procedure → checked transitions and runbooks. Stopping early → the transition check simply says "preconditions not met" and you cannot leave. Failing to reactivate old lessons → versioned practices parked outside the model. One countermeasure per failure mode.
The paper compresses this into a single claim: StateM "turns selected postmortem findings into persistent, executable preconditions and practices" (Abstract). In other words, don't write the insight into a doc — embed it as a condition that gets checked at runtime. The arXiv comment field reads "Harness Scaling, Semi-Self-Evolving Agent," and this accumulation is presumably where the semi comes from: it evolves, but in a form a person can still audit.
Comments
Sign in to comment