Paper Explained: When EOS Tokens Disagree — Why On-Policy Distillation Students Forget How to Stop

In on-policy distillation, student responses can grow until they hit the generation budget. This paper (arXiv:2609.20511) traces a major cause to student and teacher marking 'the end' with different special tokens. We walk through it from scratch, using only the paper itself.

When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

The student who got the answer right and kept talking

A model solving a math problem writes the correct answer at token 1,094. Then it doesn't stop. It repeats itself for another 7,098 tokens until it hits the generation budget — 86.6% of the response is dead weight. That's not a made-up example; it's a real training rollout from the paper's appendix (Appendix F.1).

The paper is "When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation" (arXiv:2609.20511, posted 2026-09-17).

Here's what the abstract claims. In on-policy distillation (OPD), student responses can become excessively long, sometimes exhausting the generation budget. The authors identify a key source: a termination-token mismatch between a pretrained (base) student and a post-trained teacher. Across Qwen3, Llama, and Gemma, the two models can put their stopping probability on different EOS tokens — even when their declared stopping sets are identical. The mismatch suppresses the student's own way of stopping without reliably teaching it the teacher's. Aligning the decoding stop set alone doesn't help, but treating functionally equivalent EOS tokens as one shared semantic "stop" action substantially reduces mismatch-induced inflation in all three families. A stage-wise study on K2-Horizon also reveals a separate, late-stage inflation that survives termination alignment. The bottom line: termination mismatch is an important source of OPD length dynamics, but not the only one.

Background: what on-policy distillation is

Think of a tutor. Classic distillation is "copy the teacher's model answer." On-policy distillation (OPD) is "you write your own answer, and the teacher marks it up token by token." The student gets corrected on exactly the kind of answer it would actually produce (§1).

Section 2 of the paper turns the red pen into a formula. Call the student $\pi_\theta$, the teacher $\pi^E$, the prompt plus the partial answer so far $s_t$, and the token the student just wrote $y_t$.

In words: how likely the teacher thinks this exact token is, minus how likely the student thought it was. Positive means "write more of this"; negative means "write less of this."

The crucial detail: the teacher only grades tokens the student actually sampled. The paper calls this sampled-token OPD, and every experiment uses it (§2). A token the student never writes almost never gets graded.

Intuition: two ways to say "I'm done"

An LLM ends its response by emitting a special token — an EOS (end-of-sequence) token. The catch is that several tokens can mean "done." Per the paper's Table 1, the Qwen3 base model declares only <|endoftext|>, while post-trained Qwen3 also stops on the chat terminator <|im_end|>. Llama 3.2 follows the same pattern. Gemma 3's PT and IT checkpoints declare the same pair, <eos> and <end_of_turn> (§2).

Picture the next-token distribution as a bar chart that sums to 1. If there are several EOS bars, the model's urge to stop can be split across them.