2026-08-29·★ MEMBER·PAPER·9 min read

On-Policy Distillation — Learning From What the Student Actually Writes

Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD.

#distillation #on-policy #exposure-bias #reinforcement-learning #kl-divergence #training

ModalitytextTasktraining

On-Policy Self-Distillation without Any Supervision

Primary source — what this article is built on

undefined2026-08-06→undefined2026-08-291 mo later

On-Policy Distillation of Language Models: Learning from Self-Generated MistakesarXiv:2306.13649Paper page·PDF

Knowledge Distillation of Large Language ModelsarXiv:2306.08543Paper page·PDF

Sequence-Level Knowledge DistillationarXiv:1606.07947Paper page·PDF

Sequence Level Training with Recurrent Neural NetworksarXiv:1511.06732Paper page·PDF

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningarXiv:1011.0686Paper page·PDF

On-Policy Self-Distillation without Any SupervisionYijiang Li, Bingyang Wang, Yijun Liang et al. · 2026-08-06 · v2arXiv:2608.06296Paper page·PDF

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement LearningZi-Han Wang, Zhengxi Lu, Zhiyuan Yao et al. · 2026-08-06 · v1arXiv:2608.05987Paper page·PDF

undefined

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).

undefined

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

The passenger seat at driving school

There are two ways to learn to drive: watch a hundred hours of video of a good instructor, or hold the wheel yourself while an instructor beside you says "you're drifting left."

The video method has a fatal hole in it: it never once corrects the mistakes you actually make. Maybe your habit is hugging the left edge — but the instructor in the video never hugs the left edge, so the one thing you most need ("when you drift left, here is how to recover") appears nowhere in those hundred hours.

Distillation has the same two modes. Turning the teacher's own text into a dataset and having the student copy it is off-policy distillation. Letting the student write its own text and having the teacher say, at every token, "what you should have put here is this" — that is on-policy distillation. "Policy" here just means whoever is generating the text: if the training sequences came from the student itself, it's on-policy; if from someone else — the teacher, an existing corpus — it's off-policy.

The basics of distillation — why a teacher's probability distribution carries more than a one-hot label, what the temperature $T$ is doing — are covered in The Math of Distillation. This article looks only at what comes after: which sentences you apply that teacher signal to.

The difference is one symbol

Write out the token-level distillation loss. Let $\pi_\theta$ be the student policy ( $\theta$ its parameters), $p_T$ the teacher's distribution, and $y_{<t}$ the first $t-1$ tokens of a sequence. Off-policy first, with the training set $\mathcal{D}$ fixed.

\mathcal{L}_{\text{off}}(\theta)=\mathbb{E}_{y\sim\mathcal{D}}\left[\sum_{t}\mathrm{KL}\big(p_T(\cdot\mid y_{<t})\,\big\|\,\pi_\theta(\cdot\mid y_{<t})\big)\right]

(1)

In words: draw a sequence from a fixed dataset, measure the gap between the teacher's distribution and the student's at every position, sum it up, and push it down. KL divergence measures disagreement between two probability distributions — zero when they match, larger the further apart they are.

Now on-policy.

\mathcal{L}_{\text{on}}(\theta)=\mathbb{E}_{y\sim\pi_{\theta_{\text{old}}}}\left[\sum_{t}\mathrm{KL}\big(p_T(\cdot\mid y_{<t})\,\big\|\,\pi_\theta(\cdot\mid y_{<t})\big)\right]

(2)

Everything inside the brackets is character-for-character identical to equation (1). The only change is the subscript on the expectation: $y\sim\mathcal{D}$ became $y\sim\pi_{\theta_{\text{old}}}$ . That is, stop drawing sequences from a dataset and let the current student write them. The $\theta_{\text{old}}$ reflects an implementation fact: the weights used for generation lag slightly behind the weights being updated. That will matter later.

The formulas look nearly the same, yet that one subscript changes the training cost, the shape of the implementation, and which failures you can fix.

Exposure bias — the road you never drove in training

Language models are autoregressive: emit a token, append it to the input, emit the next. But during off-policy training, what gets appended is the token the teacher (or the ground-truth data) wrote, never the token the student produced. This is teacher forcing.

At deployment the student eats its own output instead. It is worse than the teacher, so somewhere it places a token that drifts — and from that moment the context it sees is a context that never appeared in training, where nobody ever told it what to emit. The next token drifts further. This spiral is exposure bias, a term from Ranzato et al.'s MIXER paper (arXiv:1511.06732).

A rough number helps. Suppose a model stays on "roads it saw in training" with 99% probability per token. Write 200 tokens and you get $0.99^{200}\approx 0.13$ — more than four generations in five leave the training distribution somewhere along the way. However good the per-token accuracy, length forces you off the map. That is why exposure bias barely registers on short replies and suddenly bites on long reasoning chains and long agent trajectories.

On-policy distillation doesn't mitigate this; it dissolves it by definition. If the sequences you train on are the student's generations, the training distribution and the deployment distribution agree from the start. The moment the student drifts left, the teacher says "right." Passenger seat, not video.

FIG 1The rollout temperature slider. Turn it down and the student only ever walks one safe path, so its own mistakes never enter the training data. Turn it up too far and you spend the teacher budget on tail tokens the model would never emit at deployment

On-Policy Distillation — Learning From What the Student Actually Writes

The passenger seat at driving school

The difference is one symbol

Exposure bias — the road you never drove in training

Members-only from here

References

Comments