2026-09-03·★ MEMBER·PAPER·11 min read

Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

A third to a half of the teacher's token-level grades are wrong, yet the student improves just as fast either way. This paper traces on-policy distillation's gains not to imitating a teacher but to suppressing the student's own low-probability tokens — and drops the teacher entirely.

#on-policy-distillation #reinforcement-learning #llm #entropy #post-training

ModalitytextTaskinference

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Primary source — what this article is built on

undefined2026-08-31→undefined2026-09-03same month

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-ImprovementYi Ding, Ruqi Zhang · 2026-08-31 · v1arXiv:2608.31046Paper page·PDF

undefined

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.

Is the student really learning from the teacher?

The paper this article covers is titled "Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement" (Yi Ding, Ruqi Zhang / Purdue University / arXiv:2608.31046 / 31 Aug 2026).

Here is the abstract in one paragraph. On-policy distillation (OPD) has attracted attention as a way to get dense, token-level supervision, in contrast to reinforcement learning with verifiable rewards (RLVR), which only hands back a sparse outcome-level signal. But the teacher in OPD is asked to grade text it would never have written itself — the student's own rollouts. The authors measure how reliable that grading actually is, and find substantial noise whose prevalence grows as the teacher gets bigger. More surprisingly, the student converges to the same performance whether that noise is kept or removed. So where do the gains come from? The authors trace them to the suppression of the student's own low-probability tokens — something that needs no teacher at all — and propose a supervision-free method, OPSA (On-Policy Self-Adaptation).

An analogy: whose homework is being graded?

Picture a tutor marking a student's homework. The tutor has a model answer in mind. In OPD, though, what gets marked is not the tutor's model answer but the student's own attempt. The tutor keeps encountering phrasings they would never have chosen, and starts grading on a simple reflex: not how I would put it, therefore wrong.

That mismatch is what the paper calls off-policy grading. The further the student's style drifts from the teacher's, the less the grades mean.

The mechanism: what OPD actually minimizes

OPD minimizes the reverse KL divergence between the student $\pi_s$ and the teacher, computed over prefixes the student itself sampled (§2.1). With the K1 estimator, the loss looks like this:

\mathcal{L}_{\text{OPD}}=-\mathbb{E}\left[\frac{1}{|y|}\sum_{i=1}^{|y|}A_{i}\log\pi_{s}(y_{i}\mid x;y_{<i})\right]

(1)

Here $y$ is the sequence the student generated, $y_i$ is its $i$ -th token, and $x$ is the prompt. $A_i$ is the advantage: how much the teacher likes that token, positive when it approves and negative when it does not. The whole expression says only one thing — push up the probability of tokens the teacher liked, push down the ones it disliked.

There is a practical catch worth naming: computing this requires white-box access to the teacher's logits and a shared vocabulary between student and teacher (§1). A teacher you can only reach through an API is off the table from the start. If reverse KL itself is unfamiliar, KL divergence from scratch covers the background.

Measured: the teacher is wrong about a third of the time

The first thing the paper does is ask how trustworthy $A_i$ really is (§2.2). Since there is no ground truth for what every token's supervision should be, the authors restrict the measurement to tokens inside the final \boxed{} answer, where a verifier can settle correctness. Supervision counts as noisy when the sign of the teacher's advantage disagrees with the verifiable reward — a positive advantage on a wrong answer, or a negative one on a right answer.

The setup: Qwen3-1.7B as the student with thinking mode disabled, and Qwen3-4B / 30B-A3B / 235B-A22B-Instruct as teachers. For each of 500 questions from DAPO-17k, they sample one correct and one incorrect student response.

The results (§2.2):

4B teacher: 20.4% of correct trajectories receive negative advantages on their answer tokens, and 40.8% of incorrect trajectories receive positive ones — an overall noise rate of 30.6%
30B-A3B teacher: 34.7%
235B-A22B teacher: 50.6%

The noise increases with teacher scale, which is the opposite of what intuition suggests. The explanation is that a stronger teacher is further from the student's distribution, so its grading is even more off-policy.

The surprise: the student doesn't care about the noise

The obvious next move is to filter the noise out. The paper actually tried it (§2.3). Trajectories were partitioned by whether they contained noisy signals, and three conditions were compared: training on everything, training only on noisy trajectories, and training only on clean ones.

All three converged to comparable accuracy after a similar number of gradient steps. Training exclusively on trajectories full of noisy advantages improved the student at roughly the same rate as standard OPD. That result sets up the paper's central question: if the supervision is this unreliable, the student's improvement may not be coming from matching the teacher's behavior at all.

FIG 1Lowering the temperature sharpens the distribution and starves the tail. OPSA, later in this article, is essentially varying how much to sharpen — position by position

So where do the gains come from?

The paper separates the question into two: which tokens matter (§3.1) and which learning signals matter (§3.2).

Looking at the gradient, learning stalls in two regimes: when $|A_t|$ is small, and when the token's probability is already high (§3.1). Measuring the actual advantage distribution shows exactly that pile-up — 29.2% of tokens get exactly zero advantage, and 51.7% have magnitude below $10^{-4}$ . Those near-zero tokens coincide almost entirely with the tokens the student emitted at high log-probability.

The decisive experiment follows. When training is restricted to the student's high-logp tokens only, AIME24 performance shows no noticeable improvement. And replacing the advantages with random values drawn from $[-1,1]$ leaves the result essentially unchanged (§3.1). High-logp tokens carry no effective learning signal, no matter what you assign to them.