When Distillation Fails — Capacity Gaps and Contagious Overconfidence

A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles.

DAPD: Dual-Anchored Policy Distillation

Why the master's apprentice does not become a master

Say you apprentice yourself to a chess grandmaster. You watch the best move played, every day, from across the board. You still may not become a grandmaster, and there are three reasons.

The first is your capacity. Behind each of the master's moves lies a calculation dozens of plies deep. An apprentice who cannot see that far memorizes the shape of the hand and settles into an averaged style.

The second is when the master is wrong. Masters misread positions confidently. The apprentice copies the wrong move with the same diligence as the right one, and copies the confidence with it.

The third is the positions. The master's games skew toward the shapes the master is good at, and the swamp the apprentice blunders into never appears there. In the place with no example to follow, the apprentice is alone.

These three map exactly onto the three ways knowledge distillation fails: the capacity gap, contagious overconfidence, and distribution shift. The mechanics of distillation are covered in Knowledge Distillation from Scratch, so here we look only at what is going wrong when it goes wrong.

The objective, stated once more

Let us be precise about what is being minimized. Distillation pulls the student's output distribution $p_S$ toward the teacher's $p_T$, and the canonical loss looks like this.

Symbol by symbol. $\mathrm{CE}(y, p_S)$ is ordinary cross-entropy against the ground-truth label $y$ — the "check your answer" term. $\mathrm{KL}(\cdot\|\cdot)$ measures how far apart two distributions are; it is the KL divergence. $T$ is the temperature: dividing the logits by $T$ before the softmax flattens the distribution. $p^{(T)}$ is that flattened distribution, $\alpha$ balances the two terms, and $T^2$ restores the gradient magnitude the temperature shrank.

In words, the loss says: match the correct answer, and also imitate how much the teacher hesitated between the alternatives. All three failure modes are side effects of that instruction.

Failure 1: the capacity gap — a small vessel returns an average

Notice first that nothing in the loss asks whether the student is able to represent what it is told to match. The loss only says "get closer," and it keeps saying it to students that cannot.

This is what bites when the student has too few parameters or too few layers. A small model cannot reproduce the teacher's fine structure, so it settles for a flattened average of it. In multiple-choice terms: where the teacher answers crisply with "A 80%, B 15%," the student returns a smeared "A 45%, B 30%, C 25%." The loss falls while accuracy stalls.

The reason for the flattening is the direction of the KL in equation (1). $\mathrm{KL}(p_T \| p_S)$ is anchored on the teacher, so the student is punished heavily wherever the teacher puts mass and the student does not, while student mass where the teacher has none is punished only lightly. Because of that asymmetry, a student without the expressive power to pick one of the teacher's peaks is pushed to smear itself across all of them. Statisticians call it mode averaging.