Paper Walkthrough: DAPD — Breaking the Teacher's "Cheat-Sheet Illusion" in Distillation with Dual Anchors

When you distill from a teacher that can see the reference solution, the student learns to act as if invisible answers were still there — the privilege illusion. DAPD (arXiv:2608.01735) traces this failure to information asymmetry and fixes it with two levels of anchoring. A walkthrough grounded strictly in the paper itself.

DAPD: Dual-Anchored Policy Distillation

What happens when the tutor teaches with a cheat sheet

Picture a math tutor teaching a student while keeping the answer key on the desk. The teaching itself is sound. But the student ends up imitating more than the reasoning — they also copy the behavior of someone who has the answers at hand. Then, stuck in the real exam, they write "I recall the answer is probably 36/7" with nothing to back it up.

The first half is a metaphor; the "36/7" is not. Figure 4 of the paper shows an actual model output on an AIME24 geometry problem: after its derivation collapses, the model says "I recall that the answer is likely 36/7" and returns a wrong answer (§4.2).

The paper we're reading, DAPD (Dual-Anchored Policy Distillation) (arXiv:2608.01735, published 2026-08-03), studies on-policy self-distillation (OPSD) — an increasingly popular recipe for LLM post-training — and identifies the root cause of this symptom, which the literature calls the privilege illusion: an information asymmetry between the teacher and the student. It then removes that asymmetry with two levels of anchoring. The paper reports gains that survive scale: +2.00 points on average across six benchmarks on Qwen3-4B, and +2.78 points even at 32B (§1).

Background: what OPSD actually trains

Distillation teaches a student model to imitate the teacher's probability distribution over the next token, not just the single correct token. That distribution carries dense information — including how the teacher hesitates — at every position.

On-policy distillation goes further: the teacher supervises at positions sampled from the student's own rollouts. Instead of correcting someone else's perfect essay, the teacher coaches the student exactly where the student actually wanders, shrinking the gap between training states and inference states.

The twist in OPSD is that the teacher is not a separate model — it is the same model shown the reference solution (§2.1). From one policy $p_\theta$, you build two distributions that differ only in what they are conditioned on. Here $x$ is the prompt, $y$ the student's rollout, $y^*$ the reference solution, and $t$ the token position:

• None distribution $p_{\mathrm{None}}=p_\theta(\cdot\mid x,y_{<t})$ — the model seeing only the prompt and its own partial output: the same condition it faces at inference
• Cross distribution $p_{\mathrm{Cross}}=p_\theta(\cdot\mid x,y^*,y_{<t})$ — the same model at the same prefix, but with the cheat sheet

OPSD uses Cross as a frozen teacher and None as the trainable student (§2.1):

Put in words: at each position of the student's own draft, pull the distribution of the model without the reference toward the distribution of the model with it. $\mathrm{D}$ is a divergence between token distributions — what to actually put there is the subject of KL divergence from scratch — $\operatorname{sg}[\cdot]$ stops gradients on the teacher side, and $|y|$ is the rollout length. So the line as a whole is a running average: score the mismatch between the two versions of the model at every step of the draft, then divide by how long the draft was.

The student has no cheat sheet at test time

Here is the structural hole: the teacher (Cross) sees privileged information — the reference — throughout training, while the student sees none of it at inference. The paper measures the resulting symptom at the level of behavior. It defines wrong claims as outputs that first admit the derivation has failed, then assert a concrete answer as if pulled from memory or a reference, and end up wrong (§2.1, §C.2).

The trend is stark. Averaged over five Qwen3 scales, as OPSD training proceeds, wrong claims rise from 13.0 to 37.0 per 10,000 generations, while the reasoning score (Avg@12 over AIME24/AIME25/HMMT25) falls from 59.56 to 53.24 (§2.1). The more thoroughly the student internalizes cheat-sheet behavior, the worse it performs when the sheet is gone — the illusion and the degradation move together.