Paper Walkthrough: HarnessDev — Can an LLM Build and Maintain the System It Runs Inside?

A ground-up walkthrough of HarnessDev, a benchmark that asks whether an LLM can build an agent harness from a deliberately useless seed and then improve it from execution feedback. Models can build one; the gains rarely survive contact with held-out tasks or a different runtime model.

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Same weights, different container, different score

With identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 but 49.6% inside Codex CLI. The paper opens with that number (§1). Nothing about the model changed. What changed was the software outside it.

That outside layer is called the agent harness: how the execution loop turns, which tools exist and when, what enters the context window, how failures are recovered from, how results get verified. Everything that turns model output into action (§1).

The paper is "HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?" (Yuhao Wu et al., arXiv:2609.01437, 1 September 2026, ByteDance Seed and collaborators).

Its argument, in brief. As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure — the harness. Changing that harness while holding weights fixed can substantially alter task performance. Yet current agent evaluations report downstream performance under one chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. So the authors introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. It has two stages. In Creation, the agent starts from a minimal seed and a handful of cases and builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, aiming to raise benchmark performance. Each constructed harness is then scored on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results span six creator LLMs, four domains, and five downstream benchmarks totalling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. The finding: generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some gains, but they are unstable and transfer only partially to held-out tasks. A fixed-runtime experiment further shows the gains depend strongly on the model executing the harness — transfer across models is limited.

Moving the unit of evaluation

Most agent benchmarks begin after the problem has been made executable: task specified, judge defined, scaffold fixed. That is necessary for controlled comparison, but it hides the work that dominates real deployment (§2). The paper points at the forward deployed engineer as the visible crystallisation of that work, and notes the three pieces of structure benchmark designers normally presuppose: the target is vague, the feedback signal is absent or unreliable, and the execution system does not yet exist in usable form. HarnessDev takes on the third layer.

The formalisation is short (§3.1).

In words: a creator model $L_C$ works inside a development environment $D$ to produce a harness $H$. $H$ is then frozen, an executor model $L_E$ runs a downstream task $x$ inside it, and an evaluator $J$ scores the output $y$. Splitting $L_C$ from $L_E$ is the whole point — otherwise "the harness was good" and "the model running it was smart" collapse into one number. Note that $D$ is used only while building, and $L_E$ appears only after the freeze.

Starting from a deliberately useless seed

Every creator receives the same $H_{\mathrm{seed}}$: a runnable compatibility layer, not a task-solving agent (§3.2). It parses task and model config, exposes permitted low-level tools, and writes the required results, trajectories and logs. That is all. It has no agent loop, no task decomposition, no tool policy, no context management, no persistent state, no verifier, no retry or recovery, no stopping rule. Left unmodified it produces an empty or partial artifact and scores zero on every one of the five downstream benchmarks. So any nonzero Creation score is entirely execution logic the creator added.

The responsibilities are organised into six modules (Appendix C): $E$ execution, $T$ tools, $C$ context, $S$ state and memory, $L$ lifecycle and recovery, $V$ verification. Nobody has to create six files with those names; what is checked is whether the finished system actually performs those responsibilities rather than describing them.

The paper is explicit about why grading a generated harness is harder than grading a generated answer. A harness can overfit to the model that wrote it, memorise the development examples, improve one capability while silently regressing another, or raise the feedback-set score through benchmark-specific tricks that do not transfer. Hence two axes, measured on held-out tasks.

Creation: they can build one, but the domain gap is severe

The six creators are Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max and Seed 2.0 Pro. The development environment is Claude Code 2.1.177, except GPT-5.5, which uses Codex 0.144.3. Three harnesses are built independently per creator–benchmark pair and results are reported as avg@3 (§4.1).

Under Self-Eval — where the creator also serves as executor — Opus 4.8 has the highest overall score at 67.8, against 86.2 for the human-engineered reference (§4.2, Table 3). That average is close to meaningless on its own, though, because the breakdown is so lopsided.