Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure Logs

A ground-up walkthrough of AutoSaddler, which automatically optimizes the harness around an LLM agent — prompts, tools, and middleware — by repeatedly diagnosing failure traces and generating structured patches. It beat the base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points.

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

Same model, different score

Two teams use the identical LLM, and their agents differ by ten points in success rate. What separates them is not the model but everything around it: the system prompt, tool implementations, tool docstrings, the control logic of the agent loop, the wording of hooks. That surrounding layer is what people call the harness.

Today's paper asks a machine, rather than a person, to build it. The original title is "AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces" (Sungho Park et al., arXiv:2608.23041, 24 August 2026).

The paper's own claim, condensed: LLM agents remain unreliable on long-horizon tasks, where small local failures compound over extended interactions and lead to overall task failure. External harnesses can substantially improve robustness, but designing one is a manual and expensive search over a large space of prompts, tool configurations, and control logic. The authors therefore formulate harness improvement as an offline learning problem and introduce AutoSaddler, which iteratively updates the harness using failure signals from mini-batches. It combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Across GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0, it improves over the corresponding base harnesses by 9.0, 9.6, and 10.0 percentage points.

What a harness actually is

A harness is horse tack. You don't have to retrain the horse — change the saddle and the reins, and the rider's intent starts arriving intact. The Saddler in the title is the person who makes that tack; the paper automates their craft.

Concretely, three things count as tack, and the paper restricts the optimization space to exactly these (§3): the prompt (system prompts and instructions), the tools (which tools exist, and their arguments, return values, and docstrings), and the middleware (runtime control logic such as hooks and agent-loop behavior). Memory and skill curation are explicitly out of scope, because the setting assumes tasks are stateless and independent.

The trouble is that the combination of these three layers is enormous, and evaluating a single candidate is expensive. The agent may run dozens of steps before success or failure becomes visible, and reading that long trajectory to find the cause is itself heavy manual work (§1).

Seeing harness design as learning

The central move is to treat harness improvement as mini-batch training (§4, Appendix A). Split the task set into train, dev, and test; each iteration, draw a small mini-batch from the training split, run the current harness, derive an update from the failures, and check on dev whether it generalizes. The outer skeleton is exactly neural network training.

What differs is the inside. Harness parameters are not numbers but text and executable code, so they are not differentiable. Without a gradient, nothing tells you which direction to move or how far. Every design decision in this paper follows from that fact.

Writing down the objective

Draw a task $x$ from a task distribution $\mathcal{T}$ and run the agent under harness parameters $\theta$; you stochastically obtain an execution trace $\tau$ and a final output $\hat{y}$. The optimization space is the triple (§3):

Put plainly: the only things you may touch are the instructions, the tools, and the runtime control — never the model weights. What you want to maximize is expected performance over the whole task distribution, with $y^{*}$ the gold answer and $\mu$ a task-level metric:

That is "the average score when you draw many different tasks and run each of them many times." The expectation is doubled because two sources of randomness are being averaged out: which task you draw (outer) and how the agent happens to execute (inner). $J(\theta)$ is never directly observable, and each execution burns a rollout, so in practice the procedure is: within a rollout budget $K$, return whichever explored candidate scores highest on dev. The test set is touched only for the final measurement.

Four sessions per iteration

Here is one iteration (§4). Run the current harness $H_n$ on a mini-batch $B_n$ and collect traces from both successes and failures. In the diagnosis-patch session, the agent receives the failure traces and the harness source code, progressively drills into the long trace to pin down a root cause, weighs alternative hypotheses, and emits a structured patch $\Delta\theta_n$. Notably, the paper deliberately does not separate diagnosis from patch generation, so that context gathered while diagnosing feeds straight into the patch.

The patch is verified on the same mini-batch; only if $\hat{J}_{B_n}(H'_n) > \hat{J}_{B_n}(H_n)$ does it earn an additional dev-set evaluation. The reflection session then compares before and after, sorting outcomes into fixed / regressed / still-failing / still-passing, and articulates why the patch worked or why it regressed. The lessons land in EvoDAG, a directed acyclic graph whose nodes are explored harnesses and whose edges are diffs. Finally the evolution session builds the next candidate $H_{n+1}$ — and it need not continue from $H'_n$. It can recombine good parts from any earlier lineage.