Designing Distillation Data — Deciding What to Ask the Teacher

What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

A great tutor can't help if the problem set is thin

Imagine hiring a tutor. Suppose you find an outstanding one. What ultimately moves the student's score is still which problems you had the tutor work through, and how many of those worked solutions the student saw. If your tutor has mastered calculus but the problem set you handed over contains no calculus, none of that knowledge reaches the student. And if you make the student grind through a hundred pages of arithmetic they already have cold, you have burned time and gained nothing.

Knowledge distillation works exactly the same way. As we saw in Knowledge Distillation from Scratch, distillation means pulling the student's output distribution toward the teacher's. The loss function and the temperature are the ones described there. But what actually determines how the student turns out is not the fine print of the loss — it is which inputs you performed that pulling on.

This article is about that set of inputs: the distillation data.

What is actually being distilled today

In classical distillation you could take the teacher's logits (the raw scores that go into softmax) directly, because both models sat on your own machine.

In modern LLM distillation, the teacher is often a huge model behind an API, or teacher and student use different vocabularies (different token inventories). Then the probability table itself is out of reach. All you get is the text the teacher generated.

That is why sequence-level knowledge distillation became the norm: have the teacher solve problems, treat its output text as the target label, and train the student with ordinary supervised learning. Kim and Rush demonstrated this in machine translation in 2016, and nearly all LLM distillation today has this shape.

Notice what happened. The whole problem of distillation turned into a data-building problem. The loss is the same one you use for ordinary language model training. All the design freedom moved into: which inputs you collect, how many times and at what temperature the teacher answers, and which answers you keep. The rest of this article is about that.

The mechanism: the student is only constrained on the problems you drew

One equation is worth pinning down. Distillation loss can be written roughly like this.

Symbol by symbol: $x$ is an input (a prompt), $q$ is the distribution you draw inputs from when building the distillation set, $p_T$ is the teacher's output probability, $p_S$ the student's, $\theta$ the student's parameters, and $\mathrm{KL}$ measures the gap between two probability distributions (KL divergence).

In plain words, the equation says: "make the teacher and student answer alike, but only on the problems $q$ picked." Flip that around and it says the equation demands nothing at all of the student on inputs $q$ never drew. Whatever the student says there is an accident that happened outside of training.

That is the starting point for data design. The teacher's intelligence sets the quality of $p_T$, but $q$ is ours to choose — and a student's weaknesses tend to be shaped exactly like the holes in $q$.