A Field Guide to Distillation Recipes — logit, feature, attention, self

Every distillation recipe is an answer to one question: which part of the teacher should the student match? This guide lines up output (logit), intermediate feature (FitNet), attention, and self-distillation in one table and four equations, then turns the choice into a decision you make from constraints — is the teacher behind an API, do the layers line up, do the head counts match.

Distilling the Knowledge in a Neural Network

There is more than one place to copy from

When you learn to cook from a master, there are several things you could imitate. You could copy only the taste of the finished dish. You could copy how the vegetables are cut during prep. You could copy where the master's eyes go while the pan is on the heat. Or you could skip the master entirely and use yesterday's version of yourself as today's standard.

That is the whole taxonomy of distillation recipes. Which part of the teacher does the student match? Change the location and you change what you need from the teacher, the shape of the loss, and the conditions under which it works at all.

The foundation — why a trained model's output carries more information than a hard label — is covered in Knowledge Distillation from Scratch. This article is the map that comes next: which recipe do you actually reach for.

The four recipes in one table

The rest of the article opens each row in turn: equation, then something you can play with, then code.

Recipe 1: output distillation — the fewest ingredients

The classic recipe, still the workhorse. You pull the teacher's and the student's final probability distributions together and never touch the teacher's internals.

Put in words, this says: mix "match the hard label" and "match the teacher's distribution after it has been softened by temperature" in proportion $\alpha$. Here $y$ is the ground-truth label, $\mathrm{CE}$ is cross-entropy, $p_S$ is the student's probability, $p_T^{(T)}$ and $p_S^{(T)}$ are the teacher's and student's distributions flattened by temperature $T$, and $\mathrm{KL}$ measures how far apart two distributions are (KL Divergence from Scratch).

Temperature divides the logits by $T$ before the softmax: raise $T$ and the distribution flattens. Flattening surfaces the runners-up — the candidates that were not first but were close — and that is where the teacher's knowledge lives. The $T^2$ in front of equation (1) is a correction. Dividing by the temperature shrinks the gradient by $1/T^2$, so without multiplying it back, changing the temperature silently changes your effective learning rate.

The strength of this recipe is how little it asks for. You do not need the teacher's weights, the architectures can be completely unrelated, and as long as the vocabulary is shared it runs. That is why it is the default first attempt.

The weakness is the mirror image: information enters at a single point, the exit. Deep students receive a faint signal in their lower layers, and whatever intermediate features the teacher had decomposed the input into cannot be recovered once they have collapsed into a probability vector. Read the next three recipes as three answers to that one problem.

Recipe 2: feature distillation — matching the prep work

If the exit alone is not enough, match the middle too. That is the FitNet idea: take the output of one teacher layer (the hint) and pull the output of one student layer (the guided layer) toward it directly.

Stated in words, all this does is move the point of comparison from the exit to the middle: pick one shelf inside each network, and penalize the difference between what sits on them. What is being compared is work in progress, not the final probability vector.