Paper Explained: JIT-Agent — A Model That Writes the Agent Harness On Demand

An agent's capability is not the model's alone. This is a ground-up walkthrough of JIT-Agent, a model trained to synthesize the four-module harness — memory, planning, action, tools — freshly for each task.

JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

The same cook, a different kitchen

Hire a brilliant cook. Is the quality of the food determined by their skill alone? Where the knives live, what is left in the fridge, how order tickets arrive, who does the washing up — if the kitchen is badly laid out, even a brilliant cook is slow and sloppy.

LLM agents work the same way. A foundation model like GPT or GLM is the cook, but around it there is always a harness: the machinery that decides what of the conversation history is kept and what is thrown away, how the next step is recorded, which tools and skills are exposed at this exact moment, how actions are executed, and what happens when one fails. What products like Claude Code and Codex actually are is not the model — it is this surrounding machinery.

The paper's framing is that agent capability "is not a property of model weights alone, but of the model–harness pair" (§1). A strong model fails behind the wrong memory, planner, or action protocol; a strong harness only unlocks capability when the model can understand and follow it. The dependency runs both ways.

Which leaves a problem. Harness design today is manual, task-specific, and fundamentally unscalable. JIT-Agent (arXiv:2608.25593) is an attempt to hand the job of writing the harness to a machine.

Cutting a harness into four parts

The first move is to stop treating a harness as free-form code and push it into a fixed template. The paper assumes every harness can be written as a tuple of four modules (§3.1).

In plain terms: a harness $\mathbf{h}$ is a bundle of four parts. $\mathbf{M}$ is memory, $\mathbf{P}$ is planning, $\mathbf{A}$ is action, and $\mathbf{F}$ is capability orchestration — deciding which tools get handed over. At runtime their dependency order is fixed as $\mathbf{M}\rightarrow\mathbf{P}\rightarrow\mathbf{F}\rightarrow\mathbf{A}$ (§3.1).

One step of execution is defined by four lines:

$$\mathcal{C}_t=\mathbf{F}(\mathcal{C}_{\tau},\mathbf{s}_t,\mathbf{v}_t,\mathbf{d}_t),\quad (\mathbf{s}_{t+1},e_t)=\mathbf{A}(\mathbf{s}_t,\boldsymbol{\tau},\mathbf{v}_t,\mathbf{d}_t,\mathcal{C}_t)$$

Read left to right. $\boldsymbol{\xi}_{<t}$ is the complete record of everything that has happened; memory turns it into the view worth looking at right now, $\mathbf{v}_t$ (history → view). Planning takes the task $\boldsymbol{\tau}$, the internal state $\mathbf{s}_t$, and that view, and produces one local directive $\mathbf{d}_t$ (view → directive). Capability orchestration then picks $\mathcal{C}_t$, the subset of tools to expose this turn, out of everything available $\mathcal{C}_{\tau}$. Finally the action module updates the state and emits $e_t$ — either a tool call or a terminal answer.

What makes this factorization useful is that existing agent designs turn out to be different ways of filling the same four slots. The paper's own examples (§3.1):

• Plain ReAct is $(\mathbf{M}_{\mathrm{full}},\ \mathbf{P}_{\emptyset},\ \mathbf{A}_{\mathrm{react}},\ \mathbf{F}_{\mathrm{all}})$ — keep the whole history, no explicit planner, expose every tool.
• Production runtimes like Codex and OpenCode are $(\mathbf{M}_{\mathrm{compact}},\ \mathbf{P}_{\mathrm{todo}},\ \mathbf{A}_{\mathrm{react}},\ \mathbf{F}_{\mathrm{all}})$ — the loop is still ReAct, but history is compressed near the context limit and an explicit todo list is maintained.
• Recursive systems such as ROMA are $(\mathbf{M}_{\mathrm{subproblem}},\ \mathbf{P}_{\mathrm{decomp}},\ \mathbf{A}_{\mathrm{rec}},\ \mathbf{F}_{\mathrm{route}})$ — spawn subagents, isolate their contexts, route different tools to each.

So the "schools of thought" in agent design largely reduce to which option you pick in each of four slots. And once four slots have independent choices, the number of combinations multiplies. Exhausting them by hand stops being realistic.

Cooked ahead (AOT) or cooked to order (JIT)

Automatic harness improvement is not new. The paper groups most prior work as Ahead-of-Time (AOT) (§1, §2): accumulate execution traces, optimize harness code, prompts, tools or control policy from them, and grow a single durable artifact in the hope it generalizes. That is powerful when the deployment distribution is stable, but it asks the optimizer to precompile a broadly useful harness before seeing the structure of the next problem (§1).

JIT-Agent takes the Just-in-Time position instead. The observation behind it is simple: different tasks want structurally different harnesses (§1).

• Wide-search tasks benefit from parallel evidence exploration.
• Terminal tasks favour a lean serial ReAct loop.
• Deep research needs working memory over retrieved evidence.
• Repo-scale coding is naturally mediated by a filesystem holding patches, tests, traces, and repo state.

And the right harness is not merely domain-dependent but instance-dependent. If that is true, generating one on the spot beats fitting one artifact to everything. That is the Model-as-a-Harness formulation: a trained meta-agent generates a task-specific harness, and an arbitrary off-the-shelf agentic LLM executes under it (§1).

Thirteen seeds — HarnessFactory

To generate, you first need exemplars. Under a shared protocol $\boldsymbol{\Pi}$ and a common execution kernel, the authors re-implemented 13 representative contemporary scaffolds: ReAct, Plan-and-Execute, ReSum, Flash-Searcher, GAM, MemoBrain, AggAgent, OAgent, AgentFold, HiAgent, DeepAgent, ROMA, and AOrchestra (§3.2). That collection is the seed bank $\mathcal{B}_0$ with $K_0=13$.

Forcing them onto one interface does two jobs. It tests whether the four-module space is expressive enough to cover the existing landscape, and it supplies reference material for the generator. As the system runs, the bank grows into $\mathcal{B}_n \supseteq \mathcal{B}_0$, each entry tagged with its task and the observed reward, latency, and cost.

Training harness intelligence in three stages

The paper splits the required ability into three (§1): ❶ adaptivity — matching the generated harness to the task; ❷ reliability — being executable at all, and recovering when synthesis fails; ❸ evolvability — turning execution feedback into stronger future harnesses. Together these define harness intelligence, acquired through a three-stage pipeline (§4).

Stage I: imitate a teacher, then prefer the cheaper win

A stronger frozen teacher $q_\phi$ receives the task, the protocol, the capability registry, and three reference scaffolds sampled from the seed bank, and writes a task-adapted harness. Only generations that pass protocol validation and execution checks are kept as supervised data (§4.1).

But protocol compliance is a floor, not a goal. Two executable harnesses can differ enormously in reward, latency, and cost. So candidates are compared under the same backbone and the same evaluation seeds, and a preference pair is recorded only when (§4.1):

$$r^+>r^-\ \land\ \ell^+\le\ell^-\ \land\ \kappa^+\le\kappa^-$$

Spelled out: reward $r$ must go strictly up, while latency $\ell$ and cost $\kappa$ must both not get worse — and at least one of the two efficiency terms must strictly improve. "Accuracy went up but it costs twice as much" never enters the preference data in the first place.