Paper Explained: LongHorizon-Harness — Long-Horizon Agent Tasks Are a State-Management Problem, Not an Execution Problem

Without touching the model at all, keeping task state outside execution and running a Manage-Execute-Audit loop dramatically improves long-horizon task performance — a walkthrough of Alibaba DreamX team's harness-design paper, from analogy to pseudocode.

LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks

The intern who pulls an all-nighter, and the team with a ledger

Hand an AI agent a job that takes hours and a familiar pattern appears: a strong start, then a collapse near the end. The paper we cover here — LongHorizon-Harness, from Alibaba's DreamX team (arXiv:2608.01964) — makes a simple claim: the collapse is caused less by the model's intelligence than by how the work is organized. Without modifying the model at all, changing only the outer framework (the harness), it reports lifting the pass rate on the computer-use benchmark WeaveBench from 51.8% to 80.7% (§3.2).

Start with an analogy. A conventional agent is an intern pulling an all-nighter. Inside one ever-growing conversation (its context), it plans, works, takes notes, and decides for itself whether things are done. Tired, it declares "probably finished" and moves on — and one wrong assumption quietly becomes a premise for every later decision.

LongHorizon-Harness turns this into a team. A coordinator maintains the whiteboard (the task ledger), a worker starts each unit of work with a completely fresh head, and a read-only inspector checks the deliverables against the real thing. The worker's "I'm done" never goes straight onto the ledger. Only facts the inspector has verified against the environment itself get written down. That single rule is the heart of the paper.

Why long tasks fall apart

The paper organizes the difficulty of long-horizon execution into three problems (§1). First, compounding errors and goal drift: small early mistakes distort later decisions, and the trajectory slowly veers away from the original objective. Second, context rot: as the history grows, relevant information becomes harder to retrieve, and performance can degrade sharply once context utilization crosses a threshold. Third, task-state loss: agents fail to keep hold of "what the requirements are, what is already done, and what has been discovered" — their own current position.

On top of this, the paper points out two structural limitations that remain in existing harnesses like Claude Code and Codex CLI (§1): (i) task execution and task-state management live in the same ever-growing context, and (ii) execution and completion assessment are coupled, so an incorrect self-assessment can be recorded as state and become a premise for subsequent decisions. These harnesses already have planning features and subagents — and still break down on long tasks, the paper argues, precisely because these two couplings survive.

The reframing: make it a state-management problem

The paper's reformulation goes like this (§2.1). Long-horizon execution is not one long conversation; it is a sequence of independently audited state transitions. The task state lives outside execution as an explicit record, updated only with facts independently verified from the environment. Across rounds, only the task state and the audit reports persist; the worker's raw interaction history is discarded every round.

The MEA loop: Manage → Execute → Audit

The implementation of this principle is the Manage-Execute-Audit (MEA) loop. Writing $S_i$ for the task state at the start of round $i$ and $V_i$ for the accumulated audit reports, the manager's entire job fits in one equation (§2.2):

In plain words: looking only at the original task $\mathcal{T}$, the current ledger $S_i$, and the pile of verified reports $V_i$, the manager produces the updated ledger $S_{i+1}$, the next move $q_{i+1}$ (one of four: execute, done, blocked, or ask the user), and the next work order $c_{i+1}$. The manager cannot touch the environment at all — it sees no screen and runs no commands. It is a role that judges from ledger and evidence alone.

The ledger holds three kinds of records — requirements, artifacts, and facts — each marked completed / pending / blocked / untrusted. A record can become completed only when clean audit evidence supports it (§2.2).

The executor is the only role allowed to modify the environment (§2.3). It launches each round in a fresh context, receiving only the original task, the current ledger, and the work order (goal, acceptance criteria, boundary constraints, and the audit reports it should consult). It comes in GUI and CLI flavors, each restricted to the tools assigned to its role. The neat implementation detail: the executor is Claude Code or Codex CLI, unmodified. A thin AgentAdapter layer supplies "context, tools, permissions, budget" from the outside without touching the backend's native agent loop.

The auditor is read-only (§2.4). It may use the executor's report as a hint, but it determines completion from evidence it gathers from the environment itself. Its report carries both a completion status (complete / incomplete / blocked) and an integrity status (clean / suspect / violation); if any mutation of protected state is detected during auditing, it is recorded as an integrity violation and that audit can no longer support a "completed" record. Even the rule "an inspection report is void if the inspector touched the goods" is enforced by the machinery.

The whole loop, as pseudocode:

state = init_state(task)              # register requirements as pending
for round in range(25):               # at most 25 rounds (§3.1)
    decision, contract = manager(task, state, audits)
    if decision != "execute": break   # done / blocked / ask
    report = executor(contract)       # fresh context, up to 1800 s
    audit  = auditor(contract, report, env)  # read-only, up to 300 s
    audits.append(audit)
    state = update(state, audit)      # only verified facts get in

Experiments: consistent gains on three benchmarks

The evaluation uses WeaveBench (114 tasks requiring combined GUI and CLI work), OSWorld 2.0 (108 desktop tasks with a median human completion time of about 1.6 hours), and the hard command-line suite Terminal-Bench 2.1 (§3.1).

In the matched comparison — same Qwen 3.7-Plus model, same Claude Code execution backend — the WeaveBench pass rate rises from 51.8% to 80.7%, improving in all eight domains (§3.2, Table 1). On OSWorld 2.0, binary completion goes from 2.8% to 8.3% (a 3× jump); on Terminal-Bench 2.1, from 69.7% to 77.2%. With Claude Opus 4.7 on a 34-task OSWorld subset, binary completion rises from 20.6% to 35.3% (Table 3) — evidence, the paper argues, that this is not a crutch for weak models but a gain that stacks on strong ones too.

What does the work, and what it costs

The token breakdown is telling (§3.3, Fig. 5). The manager consumes only 2–8% of total tokens — explicit state management is nearly free. The real added investment is auditing, at 19–38%. And the total cost is not a fixed multiplier: 2.3× on WeaveBench and 3.6× the output tokens on OSWorld, but 24% fewer tokens on Terminal-Bench 2.1 while scoring higher — fewer retries can more than pay for the audits.

The most striking result is the 17-task WeaveBench Games subset (Table 4). Qwen 3.7-Plus wearing LongHorizon-Harness averages 0.733 — beating Claude Opus 4.7 on plain Claude Code at 0.680. Tokens move in opposite directions: Opus drops from 16.5M to 11.1M per task, Qwen rises from 10.7M to 34.3M. From this the paper concludes that agent capability is a property of the model–harness system, not of the model alone.

The effect is task-dependent (§3.3). Gains are largest where progress can be expressed as verifiable environment state — Terminal-Bench system administration improves from 0.593 to 0.889. Where the bottleneck is a single model capability like visual precision or mathematics, gains shrink and some categories regress (data-science: 0.792 → 0.667). Auditing can detect a wrong result and trigger recovery, but it cannot supply a capability the model doesn't have.

The case studies are vivid (§3.4). Plain Claude Code retries the same unresponsive Wireshark dialog for over 400 steps and finishes at 0.59; LongHorizon-Harness externalizes the failure and the missing evidence into the ledger, so the next executor goes straight to what's unresolved and scores 0.92. In a document task, the plain run edits the XML directly, producing a visually plausible result that scores 0.00 because the prescribed workflow wasn't followed — while the harness's auditor verifies all 15 heading styles at the XML level and scores 0.89.

Limitations, and how to read the numbers

The paper itself flags several caveats. Its own WeaveBench runs use root privileges inside the task VM, while official results use a regular user account, so comparisons against official scores are reference points only (Table 1 note). On OSWorld 2.0 the method runs with a hybrid GUI+CLI tool pool, unlike the GUI-only official baselines (§A.2). And budget limited the self-run Terminal-Bench configurations to three (§3.2).

A principled limitation appears in the appendix (§B.2.3): when the true success condition depends on hidden thresholds or evaluation conventions, a misread contract can pass audit anyway — what the paper calls a "confidently verified wrong answer." And with weaker models the cost can balloon (OSWorld output tokens: 28.9K → 104K), so this is not a machine that makes everything cheaper.

How this is used in practice

For anyone building agent products or running long jobs through Claude Code / Codex CLI, these are design principles you can import today.

• Split executor and verifier subagents. Give the verifier only read-only tools such as Read / Grep (via the tool allowlist) and never write permissions.
• Record progress from environment facts, not self-reports. Adopt the rule that a TODO or plan file may be updated only after checking test results, actual files, or logs.
• Budget each role separately. The paper uses 1800 s for execution, 300 s for manage/audit, and at most 25 rounds (§3.1). With caps in place, a runaway failure loop degrades into a bounded failure that leaves a ledger behind.

Three pitfalls cause real incidents. (1) Recording the executor's "done!" directly as progress — exactly the document task that scored 0.00. (2) Giving the verifier write access — it can then break the environment mid-inspection without anyone noticing. (3) Deploying it on short tasks without pricing in the audits — auditing alone takes 19–38% of tokens, and the benefits concentrate in long, dependency-heavy tasks.

Related articles

• LLM Agents Basics — the basic tool-use loop this paper builds on
• Understanding the KV Cache — the mechanics of why growing context gets expensive
• Paper Explained: SWE-Bench ProMax — a deep dive into how agent benchmarks are designed