2026-08-29·★ MEMBER·PAPER·10 min read

Distilling Agents — How to Compress a Long Trajectory

When you distill an agent, the unit of learning changes from a single question-and-answer to an entire game. Tens of thousands of tokens come back with one bit of grading. This article works through how that gap is bridged — turn-level credit assignment, trajectory filtering and on-policy correction, and the inheritance of tool use — and maps the recent long-horizon agent literature onto those three axes.

#distillation #agents #trajectory #credit-assignment #tool-use #training

ModalitytextTaskagent

ReAct: Synergizing Reasoning and Acting in Language Models

Primary source — what this article is built on

undefined2022-10-06→undefined2026-08-293y 11mo later

ReAct: Synergizing Reasoning and Acting in Language ModelsShunyu Yao, Jeffrey Zhao, Dian Yu et al. · 2022-10-06 · v3arXiv:2210.03629Paper page·PDF

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningarXiv:1011.0686Paper page·PDF

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesarXiv:2305.02301Paper page·PDF

FireAct: Toward Language Agent Fine-tuningarXiv:2310.05915Paper page·PDF

AgentTuning: Enabling Generalized Agent Abilities for LLMsarXiv:2310.12823Paper page·PDF

Toolformer: Language Models Can Teach Themselves to Use ToolsarXiv:2302.04761Paper page·PDF

On-Policy Distillation of Language Models: Learning from Self-Generated MistakesarXiv:2306.13649Paper page·PDF

ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLarXiv:2402.19446Paper page·PDF

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement LearningZi-Han Wang, Zhengxi Lu, Zhiyuan Yao et al. · 2026-08-06 · v1arXiv:2608.05987Paper page·PDF

ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit AssignmentYijun Lu, Rui Ye, Jiajun Wang et al. · 2026-08-05 · v1arXiv:2608.05102Paper page·PDF

undefined

While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines, as well as improved human interpretability and trustworthiness over methods without reasoning or acting components. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generates human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. On two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples. Project site with code: https://react-lm.github.io

undefined

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

undefined

Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer. However, existing methods for training these agents typically treat all steps within a trajectory uniformly during both supervised fine-tuning (SFT) and reinforcement learning (RL), failing to distinguish useful actions from erroneous or redundant ones. In this paper, we propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents by converting sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions. Specifically, given a potentially obscure query and its corresponding ground-truth answer, ABC first performs Answer-Backtracked Clue Recovery, which traces back from the answer to recover intermediate clues required to solve the question. It then applies Clue-Anchored Step Scoring to evaluate each search step against these clues, converting sparse binary outcome supervision into dense step-level rewards. Based on these rewards, we develop ABC-SFT, which reweights the loss of each turn, and ABC-GRPO, which uses the step-level scores as rewards in GRPO. Building on this framework, we train ABSeeker based on Qwen3.5-4B with only 8.5k examples. ABSeeker achieves 37.3% on BrowseComp and 39.1% on BrowseComp-ZH. With context management, the scores further improve to 55.3% and 52.9%, respectively, significantly outperforming same-scale (4B) agents and even matching the performance of larger ones (approximately 30B). These results demonstrate the effectiveness of answer-backtracked step-level credit assignment for training long-horizon search agents.

A Game Record Is Not a Teacher

Suppose you have the score sheet of a game played by a chess grandmaster: a hundred moves, first to last, laid out in order. You hand it to a student and say "play exactly this." Does the student become a grandmaster?

No, for two reasons. First, the sheet never says why each move was chosen. What is written down is the outcome; the reading that produced it lived only in the player's head. Second, chess has an opponent. The moment your opponent plays something different on move three, the remaining ninety-seven moves are wastepaper.

Distilling an LLM agent puts you in exactly this position. Distillation so far has worked on question-and-answer pairs: there is an input, there is a teacher's output distribution, and you pull the student toward it. An agent, by contrast, plays dozens of moves before it answers, searching and running code along the way, with each result changing the next move. The unit of learning shifts from a question to a game.

You need no background for what follows. It is enough to picture the loop covered in LLM Agents from Scratch: think, use a tool, look at the result, think again.

The Trajectory as a Unit

An agent's game is called a trajectory. It looks like this.

\tau = \big(s_0,\ a_1,\ o_1,\ a_2,\ o_2,\ \dots,\ a_H,\ o_H,\ r\big)

(1)

Symbol by symbol: $s_0$ is the task as handed over ("fix this bug"), $a_t$ is what the agent wrote on move $t$ , $o_t$ is what the environment returned, $H$ is the number of moves (the horizon), and $r$ is the score at the end.

Note that $a_t$ is not a single token. In a ReAct-style format it is a block of several hundred tokens — a chunk of reasoning plus a tool call — and $o_t$ is a search result or a stack trace pasted back verbatim, so it runs from hundreds to thousands of tokens. At twenty moves, one trajectory comfortably reaches tens of thousands of tokens.

And that final $r$ is usually a single bit. Did the tests pass, or not. Tens of thousands of tokens written, one tick or cross returned. That gap is where every difficulty in agent distillation comes from.

Errors Multiply, They Do Not Add

If each move is independently correct with probability $p$ , the chance of getting through the whole thing is:

P(\text{success}) = p^{H}

(2)

Read it as "take the per-move accuracy and multiply it by itself once for every move." Because it multiplies rather than adds, it bites harder than intuition suggests. An agent that is 95% correct per move still fails two times in three over twenty moves — $0.95^{20} \approx 0.36$ — and $0.95^{50} \approx 0.077$ over fifty. A small gain in per-move accuracy is an order-of-magnitude gain end to end.

Reality is worse than that, because failures are not independent. One wrong turn puts every subsequent state outside the training data, and the model goes further astray. This self-amplifying drift is old news in imitation learning: the DAgger paper (Ross et al., 2011) showed that naively copying an expert's trajectories can let error compound quadratically in the horizon, and that querying the expert on states the learner actually visits brings it back down to linear. Nearly every trick in agent distillation is an answer to this one problem.

Which Move Actually Mattered?

So: you have one tick or cross, and twenty moves to spread it across. This is credit assignment. Write the distillation loss as a turn-weighted sum:

\mathcal{L} = \sum_{t=1}^{H} w_t \cdot \mathrm{KL}\Big(\pi_{\text{teacher}}(\cdot \mid s_t)\ \Big\|\ \pi_{\text{student}}(\cdot \mid s_t)\Big), \qquad \sum_{t} w_t = 1

(3)

This says: at each turn, measure how far apart the teacher's and the student's distributions over moves are (the KL), then blend those per-turn gaps using weights $w_t$ . Here $\pi$ is a policy — "in this state, with what probability does it play each move" — and KL is a yardstick for how different two distributions are, in this case how much the teacher and student disagree about what to play. The only new object is $w_t$ .

And $w_t$ can be written as a softmax over per-turn "how much this mattered" scores $z_t$ :

$w_t = \frac{\exp(z_t / T)}{\sum_{t'} \exp(z_{t'} / T)}$

$T$ is a temperature. Raise it and the weight spreads evenly across every turn; lower it and the weight concentrates on the moves that mattered. Push it high enough and you get $w_t = 1/H$ — uniform credit, which is just memorising every turn equally, ordinary supervised fine-tuning.

FIG 1Read the bars as "how much each turn mattered" and move the temperature. High temperature spreads credit thinly and evenly across all turns (rote copying); lower it and a few turns rise above the rest. How far you can sharpen before training breaks is the heart of this article

Distilling Agents — How to Compress a Long Trajectory

A Game Record Is Not a Teacher

The Trajectory as a Unit

Errors Multiply, They Do Not Add

Which Move Actually Mattered?

Members-only from here

References

Comments