Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of Steps

Trained on a single query, on-policy distillation still improves for hundreds of steps and recovers most of full-data OPD's gain. The paper explains this with state coverage and absorption rate, and concludes OPD is data-overfed but algorithm-starved.

Rethinking On-Policy Distillation of Large Language Models II: One Training Example

What happens when you cut the training set down to one question

The original title is "Rethinking On-Policy Distillation of Large Language Models II: One Training Example" (arXiv:2609.04172, published 2026-09-03, cs.AI / cs.CL, 29 pages and 20 figures). The authors are Zixuan Fu, Bingxiang He and colleagues, centred on Tsinghua University.

Here is the paper's claim up front. On-policy distillation (OPD) pairs rollouts the student generates itself with dense, token-level supervision from a teacher. Prior work has studied mostly its algorithmic behaviour, leaving the role of the training data unclear. So the authors push the data down to its minimum and train on a single query. The model keeps improving for hundreds of steps and recovers most of full-data OPD's gain, across task domains and model families. The paper explains this through the states the run visits and the rate at which the student aligns with the teacher. A single query already reaches 71.5% of the states full-data OPD visits; adding semantically distinct queries raises coverage until 16 of them reach 98.9% and match full-data training. Meanwhile alignment slows at much the same pace whether you train on one query or the whole set. OPD is therefore data-overfed but algorithm-starved.

That is an uncomfortable finding in practice, because it questions how much of a post-training budget belongs in data collection. Let's work through it.

First: what on-policy distillation actually does

Start with a comparison. Ordinary knowledge distillation is like copying the teacher's model answers into your notebook. The student never makes its own mistakes, so it never gets taught about the traps it personally falls into.

OPD inverts that. The student solves the problem itself, and at every token of its own answer the teacher writes in what it would have said, and with what probability. Correction lands only on the positions the student actually stood in, and it lands as a full distribution rather than a single label. The paper calls these two properties local and dense (§2.2).

One term to fix. A state here is the autoregressive context: the input $x$ plus the response tokens generated so far. Every generated token creates one new state.

The OPD objective is a per-token KL divergence on the visited states (§2.2).

In words: at every position $s_i$ of a sentence the student $\pi_\theta$ wrote itself, push the student's next-token distribution onto the teacher's, $\pi_T$. Here $\mathbb{E}$ is an average, $L$ is the response length, and KL divergence measures how far apart two probability distributions are.

In practice it is estimated from the single token that was actually emitted (Eq. 2).

Which says: take how confidently the teacher would pick that token and subtract how confidently the student did. Positive means the student should have emitted it more; negative, less. The paper also uses a second estimator (Eq. 3) that keeps the student's top-$k$ tokens at each state and weights them by the student's own probability, with $k=16$ for the mathematics runs (§3.1).

How far does one question actually get you?

The setup is in §3.1. For mathematics the student is DeepSeek-R1-Distill-Qwen-1.5B, the teacher is JustRL-1.5B, and the training set is DAPO-Math-17K. Evaluation is MATH-500, AMC 2023 and AIME 2025 at avg@16. The one-shot queries are chosen by the untrained student's pass rate over 8 rollouts: easy at 8/8, medium at 4/8, hard at 0/8.

The results (§3.2). Averaged over the three benchmarks, one-shot OPD reaches 68.5 at step 300 against 69.8 for full-data OPD. That is 69% of the initial teacher–student gap and 87% of full-data OPD's gain. Past step 300 the two curves stay within a band of about 3 points, and at step 1000 one-shot reaches 68.4 against 72.1, recovering 72% of full-data OPD's gain.

It holds across families too (§3.2, averaged over MATH-500 and AMC 2023). R1-Distill-1.5B goes 77.1 → 85.5, Llama-3.2-3B-Instruct 28.2 → 40.2, and OLMo-3-7B-Instruct-DPO 70.8 → 82.4. Across domains, one-shot OPD recovers 73% of the teacher–student gap on code generation, 66% on instruction following, and 64% on agentic tool use.

More striking: it works on a query the student never solves. The hard query is not solved once in 300 steps, yet its benchmark gain is comparable to the easy and medium ones (§3.2, Appendix A.1). Tightening the response-length cap and lowering the rollout temperature both leave the gain intact.

Learning from a problem you can't solve is exactly what separates this from RL with verifiable rewards. We'll come back to it.

That is the phenomenon. What follows is the paper's explanation of it — state coverage on the data side, the absorption rate on the algorithm side.