Knowledge Distillation from Scratch — Copying a Big Model into a Small One

No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.

Distilling the Knowledge in a Neural Network

The label doesn't carry what the teacher knows

Picture a four-way image classifier: dog, cat, truck, airplane. A photo of a dog carries the label "dog" — as numbers, $(1, 0, 0, 0)$. That is the entire contents of the dataset for that image.

Now show the same photo to a large, well-trained model. The output might be $(0.90,\ 0.09,\ 0.005,\ 0.005)$: it's a dog, though a cat isn't out of the question, and it's certainly not a vehicle. The ratios buried in there — dog:cat at roughly 10:1, dog:truck at 180:1 — appear nowhere in the label. Yet they encode something real that the model absorbed from millions of examples: which categories resemble each other and which don't.

Hinton and colleagues called this dark knowledge. Knowledge distillation is the business of transferring it from a large model (the teacher) into a small one (the student).

The analogy: copying answers vs. copying hesitation

Two students work through the same problem set.

The first looks only at the answer key: question 3 is C. Nothing more. The second also reads the grader's note: the answer is C; a lot of people were torn between B and C; A and D are nowhere close.

The second student extracts far more from the identical set of problems, because "B and C are easy to confuse" is information that exists independently of the correct answer. That is the whole intuition, and it cashes out in two practical ways.

More information per example. A one-hot label transmits a single class index; a teacher's output transmits a number for every class. Training converges on less data, and unlabeled data becomes usable — just run the teacher over it and you have soft labels.

It acts as a regularizer. Teaching "definitely dog, everything else exactly zero" pushes the student toward overconfidence. "Probably dog, cat is plausible" doesn't. It's label smoothing, except the smoothing comes from the teacher's judgment rather than a constant.

The mechanism: blurring the distribution with temperature

There's a catch in using the teacher's output directly. A well-trained model tends to output something like $(0.99,\ 0.009,\ 0.0009,\ \dots)$, and the very thing we wanted — whether cat or truck is more dog-like — sits in digits so small that they barely register in the gradient.

The fix is a temperature-scaled softmax: divide the logits (the raw scores that go into the softmax) by $T$ before normalizing.

Here $z_i$ is the logit for class $i$, $T$ is a positive number called the temperature, and $p_i(T)$ is the resulting probability. At $T = 1$ this is the ordinary softmax. Raising $T$ shrinks the gaps between the exponentiated terms, flattening the distribution; pushing $T$ toward zero collapses it into a one-hot vector where only the argmax survives.

Temperature, then, is a dial for how much uncertainty you make the teacher admit. Turn it up to around $T = 4$ and $(0.99, 0.009, \dots)$ relaxes into something like $(0.6, 0.2, 0.13, 0.07)$ — the buried ranking becomes a training signal with real magnitude.

The loss: mixing two objectives

The student is told two things at once: match the ground-truth label, and match the teacher's distribution.

The first term is ordinary supervised learning: $y$ is the true label, $p^S(1)$ is the student's output at temperature 1, and $\mathcal{L}_{\mathrm{CE}}$ is cross-entropy. The second is the distillation term, where $p^T(T)$ and $p^S(T)$ are the teacher's and student's distributions at temperature $T$, and $\mathrm{KL}$ measures how far apart two distributions are (covered in KL Divergence from Scratch). The mixing weight $\alpha$ is often set high — 0.9 is common — to lean on the teacher.

In plain words: keep checking your answers, but at the same time keep closing the distance to the teacher's blurred distribution.