#terminal-bench
4 articles
01
·Agents·★ MEMBER·PAPER·12 min read
Paper Walkthrough: HarnessDev — Can an LLM Build and Maintain the System It Runs Inside?
A ground-up walkthrough of HarnessDev, a benchmark that asks whether an LLM can build an agent harness from a deliberately useless seed and then improve it from execution feedback. Models can build one; the gains rarely survive contact with held-out tasks or a different runtime model.
02
·Agents·★ MEMBER·PAPER·15 min read
Paper Walkthrough: PILOT in the Loop — Fixing the Run While It Is Still Running
Agent self-improvement that only starts after execution ends arrives too late. A ground-up walkthrough of PILOT, a supervisor–worker harness that redirects the live run while distilling skills from it, from intuition to equations to measured numbers and stated limits.
03
·Agents·★ MEMBER·PAPER·11 min read
Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure Logs
A ground-up walkthrough of AutoSaddler, which automatically optimizes the harness around an LLM agent — prompts, tools, and middleware — by repeatedly diagnosing failure traces and generating structured patches. It beat the base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points.
04
·Agents·★ MEMBER·PAPER·11 min read
Paper walkthrough: StateM — 95.3% on Terminal-Bench 2.1 and a USD 15 run, without touching a single weight
Long-horizon agents fail even when the model underneath can solve every individual step. StateM leaves the weights alone and hardens only the execution system around the agent, reporting 95.3% on Terminal-Bench 2.1 and a final scoring run billed at roughly USD 15 instead of USD 574.68. A ground-up walkthrough of the harness-scaling bet.