Paper Walkthrough: LoopArena — Benchmarking the Model That Steers a Coding Agent

A ground-up walkthrough of LoopArena, a benchmark that scores the outer loop steering a coding agent while holding the agent itself fixed — including how to read the headline result of 24.69% strict success on full tasks.

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

Nobody was measuring the thing that steers the loop

Hand a coding agent a long job and your role changes. You stop writing code and start deciding what it should do next: run the tests, keep implementing, or call it done. Rather than typing each of those decisions by hand, people are building loops that monitor progress, assign work, run checks, and decide when to stop. That practice has picked up a name: Loop Engineering.

The awkward part is evaluation. When a run succeeds, you cannot tell from the final result whether the guidance was good or the coding agent was simply strong enough to carry the task on its own.

The paper this article walks through is titled LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering (arXiv:2608.28281, published 2026-08-28, DreamX Team / Alibaba Group and collaborators).

Its abstract, in short: a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit — yet the outcome of a single end-to-end run cannot separate the loop's guidance from the coding agent's ability. So the authors introduce LoopArena. The model under evaluation is the Controller: after each coding round it receives a structured summary of the run and tells a separate, fixed coding agent — the Worker — what to do or verify next, or decides to stop. Three settings differ in execution scope and cost. On full tasks the best observed Strict Success Rate is 24.69%. Across Controllers the paired reduction in estimated inference cost averages 64.4%, and the cheaper setting reproduces a similar ordering under the main Core criterion (Spearman's $\rho = 0.9747$). Data and evaluation code are released publicly.

The analogy: a site foreman with no tools

Picture a construction site. The tradesperson (the Worker) holds the tools and actually paints the wall and runs the pipe. The foreman (the Controller) holds no tools at all. All the foreman can do is say things: "do this section next," "put water through that pipe and check for leaks," "we can hand it over now."

And the foreman never looks at the site directly. What they read is a report written by a third party — the Reporter. The report covers four things: what was asked for, what was actually done and where things stand, what was checked and what that established, and what remains unresolved. It deliberately does not say what should happen next. Deciding the next move is the foreman's job.

LoopArena scores only the foreman (§1). The tradesperson, the tools, the budget, and the acceptance criteria are all held fixed; only the Controller model is swapped. That is what makes "was the agent smart, or was the instruction smart?" an answerable question.

Three roles: Worker, Reporter, Controller

Worker. The only component that can touch coding tools. Within its assigned segment it follows its native ReAct loop to inspect the repository, modify it, and run checks. Its conversation persists across the whole run (§2.2).

Reporter. Created fresh at each handoff from a copy of the Worker's conversation, using the same model configuration, with read-only access to the workspace. It writes four fields — task_context_and_constraints, work_history_and_current_state, verification_and_evidence, open_issues_and_uncertainty — and cites the relevant Worker turns as [E12]. It describes the state and does not decide what happens next (Appendix 11.1).

Controller. The model under evaluation. It reads an Evidence Packet — the harness's deterministic rendering of the Reporter's report plus the cited Worker turns — and returns a Loop Contract. It has no repository access and no coding tools; it can affect the task only through the instructions it sends.

Why does this separation matter so much? Because these runs are long. In the paper's measurements a single full-task run averages 139.81–288.90 Worker ReAct turns and 8.60–13.46 control cycles (Table 1). That is a lot of opportunities to steer wrong.

The control cycle, as a formula and as code

For task $i$ at control cycle $k$, the Loop Contract the Controller returns is (§2.2):

Here $\pi$ is the Controller model, $x_{i,k}$ is the Evidence Packet at that moment, $h_{i,k}$ is the conversation history before this decision (earlier Packets and Contracts), and $c_{i,k}$ is the Contract. In words: read the latest report plus what has happened so far, and write one instruction sheet.

A Contract carries an action — advance, verify, or stop — and a rationale. For advance or verify it also carries a worker_instruction (goal, context, required_outcomes, prohibited_actions, completion_condition), plus protected_invariants and a verification_acceptance_condition (Appendix 11.1). The harness validates the Contract and, if work continues, renders it as the Worker's next turn with no additional model call; on stop it submits the current workspace to the evaluator.

while True:
    worker.run_segment()                    # only this role holds tools
    report = Reporter(copy(worker.history)) # four-field factual report
    packet = format_packet(report, cited_turns, budget)
    contract = controller(packet, history)  # the thing being evaluated
    if contract.action == "stop":
        return evaluator(workspace)
    worker.push(render(contract))           # contract becomes the next prompt