Paper Walkthrough: DAPD — Breaking the Teacher's "Cheat-Sheet Illusion" in Distillation with Dual Anchors

When you distill from a teacher that can see the reference solution, the student learns to act as if invisible answers were still there — the privilege illusion. DAPD (arXiv:2608.01735) traces this failure to information asymmetry and fixes it with two levels of anchoring. A walkthrough grounded strictly in the paper itself.

DAPD: Dual-Anchored Policy Distillation

What happens when the tutor teaches with a cheat sheet

Picture a math tutor teaching a student while keeping the answer key on the desk. The teaching itself is sound. But the student ends up imitating more than the reasoning — they also copy the behavior of someone who has the answers at hand. Then, stuck in the real exam, they write "I recall the answer is probably 36/7" with nothing to back it up.

The first half is a metaphor; the "36/7" is not. Figure 4 of the paper shows an actual model output on an AIME24 geometry problem: after its derivation collapses, the model says "I recall that the answer is likely 36/7" and returns a wrong answer (§4.2).

The paper we're reading, DAPD (Dual-Anchored Policy Distillation) (arXiv:2608.01735, published 2026-08-03), studies on-policy self-distillation (OPSD) — an increasingly popular recipe for LLM post-training — and identifies the root cause of this symptom, which the literature calls the privilege illusion: an information asymmetry between the teacher and the student. It then removes that asymmetry with two levels of anchoring. The paper reports gains that survive scale: +2.00 points on average across six benchmarks on Qwen3-4B, and +2.78 points even at 32B (§1).

Background: what OPSD actually trains

Distillation teaches a student model to imitate the teacher's probability distribution over the next token, not just the single correct token. That distribution carries dense information — including how the teacher hesitates — at every position.

On-policy distillation goes further: the teacher supervises at positions sampled from the student's own rollouts. Instead of correcting someone else's perfect essay, the teacher coaches the student exactly where the student actually wanders, shrinking the gap between training states and inference states.

The twist in OPSD is that the teacher is not a separate model — it is the same model shown the reference solution (§2.1). From one policy $p_\theta$, you build two distributions that differ only in what they are conditioned on. Here $x$ is the prompt, $y$ the student's rollout, $y^*$ the reference solution, and $t$ the token position:

• None distribution $p_{\mathrm{None}}=p_\theta(\cdot\mid x,y_{<t})$ — the model seeing only the prompt and its own partial output: the same condition it faces at inference
• Cross distribution $p_{\mathrm{Cross}}=p_\theta(\cdot\mid x,y^*,y_{<t})$ — the same model at the same prefix, but with the cheat sheet

OPSD uses Cross as a frozen teacher and None as the trainable student (§2.1):

Put in words: at each position of the student's own draft, pull the distribution of the model without the reference toward the distribution of the model with it. $\mathrm{D}$ is a divergence between token distributions — what to actually put there is the subject of KL divergence from scratch — $\operatorname{sg}[\cdot]$ stops gradients on the teacher side, and $|y|$ is the rollout length. So the line as a whole is a running average: score the mismatch between the two versions of the model at every step of the draft, then divide by how long the draft was.

The student has no cheat sheet at test time

Here is the structural hole: the teacher (Cross) sees privileged information — the reference — throughout training, while the student sees none of it at inference. The paper measures the resulting symptom at the level of behavior. It defines wrong claims as outputs that first admit the derivation has failed, then assert a concrete answer as if pulled from memory or a reference, and end up wrong (§2.1, §C.2).

The trend is stark. Averaged over five Qwen3 scales, as OPSD training proceeds, wrong claims rise from 13.0 to 37.0 per 10,000 generations, while the reasoning score (Avg@12 over AIME24/AIME25/HMMT25) falls from 59.56 to 53.24 (§2.1). The more thoroughly the student internalizes cheat-sheet behavior, the worse it performs when the sheet is gone — the illusion and the degradation move together.

The cause is not teacher quality — it's information asymmetry

Prior work tried to filter or reweight the privileged signal. The paper diagnoses this as structurally insufficient: however you massage the signal, a teacher that holds privileged information still directly supervises a student that doesn't (§1, §2.2).

To test the diagnosis, the paper introduces a third distribution:

• Self distribution $p_{\mathrm{Self}}=p_\theta(\cdot\mid x,y,y_{<t})$ — the model conditioned on the very completion it is predicting, in full

Keep the teacher (Cross) untouched and swap the student side from None to Self, and both sides now see one full completion — the information is matched. This single intervention (named the Privileged Anchor) cuts late-stage wrong claims by 45% and lifts Avg@12 by +6.22 points (§2.2). Since the teacher construction never changed, the culprit is the asymmetry, not teacher quality.

What makes Self so useful is that, unlike Cross, it is trainable (it shares parameters with None, so gradients flow), while at the same time it is information-matched with Cross. It can stand in the gap between "the student under inference conditions" and "the teacher with the cheat sheet" (§2.3).

The DAPD design: two levels of anchoring

DAPD systematizes this bridge with two components (§3).

Dual-Path Anchoring (DPA) pins down one guidance direction (say, the reference guiding the rollout) along two paths (§3.2). Its building blocks are three directed losses, each read "teacher → student" (§3.1):

• $\mathcal{L}_{\mathrm{ent}}$ (Entangled Distillation): the original OPSD term, Cross → None
• $\mathcal{L}_{\mathrm{infer}}$ (Inference Anchor): None → Self — anchoring Self to the inference-time distribution
• $\mathcal{L}_{\mathrm{priv}}$ (Privileged Anchor): Cross → Self — distillation where both sides hold privileged information

The unconditioned path combines two of them (with $s$ the completion being taught on, $\bar{s}$ the other one):

Read it as a two-term recipe which says: keep the original OPSD term, and additionally anchor Self to the inference-time None on the opposite completion. The superscripts are bookkeeping for which finished answer the term is computed on — $s\rightarrow\bar{s}$ means "teach on this completion, anchor on the other one." Because Self and Cross share parameters and a similar privileged condition, they act as a proxy bridge, so optimizing these two terms jointly pulls the two None distributions — the alignment you actually want — toward each other. Appendix B formalizes this as an explicit upper bound built from the triangle inequality and Pinsker's inequality (Proposition 1). The privileged path is simply $\mathcal{L}_{\mathrm{priv}}$, aligning the two sides directly when both hold privileged information.

Dual-Source Anchoring (DSA) then applies DPA in both guidance directions (§3.3). References are reliable but may sit far from what the student can actually produce (off-policy); rollouts are reachable but may be wrong. DSA balances the two with a coefficient $\lambda$:

That is a weighted sum of "guided by the reference" and "guided by the model's own rollout," with $\lambda\in[0,1]$ setting the balance. Put in words, $\lambda$ is a dial: turn it to 1 and the curated solution does all the teaching, turn it to 0 and the model's own attempt does, and any value in between splits the job between them.

Experiments: gains that don't evaporate with scale

The paper trains Qwen3 models from 1.7B to 32B with LoRA on OpenThoughts data and evaluates six benchmarks: reasoning (AIME24/25, HMMT25), coding (LiveCodeBench v5, BFCL v3), and instruction following (IFBench) (§4.1).

• Qwen3-4B: DAPD reaches the best six-task average of 57.34 — +2.00 over OPSD, +1.09 over Purified OPSD and +3.85 over DOPD, the two baselines that modify the privileged signal (§4.2, Table 1)
• Scale: OPSD's gain over the base policy shrinks from +5.19 at 1.7B to +1.39 at 4B and at most +0.28 from 8B through 32B, while DAPD retains +2.41/+2.13/+3.06 at 8B/14B/32B (§4.2). The paper's reading: larger models produce rollouts with genuinely useful reasoning, but OPSD's teacher can bypass those steps using the reference — so the cost of the illusion grows with scale
• Out of distribution: a model trained only on reasoning data still gets the best OOD average of 49.64, including +4.82 over OPSD on LCB v5 (Table 2)
• Behavior: over training steps 250–300, DAPD cuts wrong claims by 73% relative to OPSD (§4.2). On the torus problem from the introduction, OPSD "recalls" its way to a wrong 43 while DAPD completes the derivation and lands on the correct 127 (Figure 4)

The ablations back the design. Each guidance source improves monotonically as paths are added — reference-guided 62.13 → 63.43 → 63.89, rollout-guided 63.33 → 64.91 → 65.09 — and combining both sources gives the best 65.28 (§4.3, Table 3a,b). The optimal $\lambda$ also drops from 0.5 at 1.7B to 0.2 at 4B and above (Table 3d): smaller models should lean on reliable references, larger ones on their own rollouts.

The most intriguing result is the reference-free variant (§4.5): replacing the curated reference with a second independent rollout already beats OPSD by +0.46 to +2.41 points, and filtering that rollout with a correctness verifier improves things further (Table 3e). The framework may not need hand-curated solutions at all.

What the paper doesn't claim

The stated limitations are worth keeping in view (§6 Limitations). (1) DAPD builds several anchored distributions during training, so training compute goes up (inference cost is unchanged). (2) The coefficients may need recalibration across scales and architectures — indeed, the best settings differ at every scale (Table 5). (3) The verified-rollout extension assumes an automatic correctness signal, which open-ended tasks may not provide. The experiments are also confined to the Qwen3 family and OpenThoughts data; other model families remain untested.

How this is used in practice

The people this paper speaks to are engineers designing post-training pipelines for reasoning models. The moment your distillation setup conditions the teacher on anything the model won't have at inference — reference solutions, tool outputs, retrieved documents — this failure mode is on the table. The paper itself lists retrieved documents, tool traces, and verifier feedback as privileged-information forms to study next (§E).

The knobs you would actually touch: the source balance $\lambda$ (paper values: 0.5 at 1.7B, 0.2 at 4B and up), the per-anchor weights $\beta$ (Table 5), the divergence implementation (the paper uses a full-vocabulary forward KL with each component clipped at 0.05 and logits divided by a temperature of 1.1, §A.6), and the LoRA setup (rank 64, scale 128, learning rate $5\times10^{-6}$, §A.6).

Two pitfalls that turn into incidents. First, benchmark scores alone will not show you the illusion forming — wire a behavioral probe into training, like the paper's wrong-claims counter that flags "derivation admitted failed, then a confident unsupported answer." Second, "make the teacher stronger" is not a safe default under information asymmetry. When a design review proposes showing the teacher more context, the first question should be: will the student have that same context at inference?

Read next

• The LoRA paper, explained — DAPD trains with LoRA (rank 64); here's how low-rank adaptation works
• Instruction tuning and RLHF — the post-training landscape this method lives in
• Loss functions and optimization — the basics of pulling distributions together with KL divergence