Designing Distillation Data — Deciding What to Ask the Teacher

What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

A great tutor can't help if the problem set is thin

Imagine hiring a tutor. Suppose you find an outstanding one. What ultimately moves the student's score is still which problems you had the tutor work through, and how many of those worked solutions the student saw. If your tutor has mastered calculus but the problem set you handed over contains no calculus, none of that knowledge reaches the student. And if you make the student grind through a hundred pages of arithmetic they already have cold, you have burned time and gained nothing.

Knowledge distillation works exactly the same way. As we saw in Knowledge Distillation from Scratch, distillation means pulling the student's output distribution toward the teacher's. The loss function and the temperature are the ones described there. But what actually determines how the student turns out is not the fine print of the loss — it is which inputs you performed that pulling on.

This article is about that set of inputs: the distillation data.

What is actually being distilled today

In classical distillation you could take the teacher's logits (the raw scores that go into softmax) directly, because both models sat on your own machine.

In modern LLM distillation, the teacher is often a huge model behind an API, or teacher and student use different vocabularies (different token inventories). Then the probability table itself is out of reach. All you get is the text the teacher generated.

That is why sequence-level knowledge distillation became the norm: have the teacher solve problems, treat its output text as the target label, and train the student with ordinary supervised learning. Kim and Rush demonstrated this in machine translation in 2016, and nearly all LLM distillation today has this shape.

Notice what happened. The whole problem of distillation turned into a data-building problem. The loss is the same one you use for ordinary language model training. All the design freedom moved into: which inputs you collect, how many times and at what temperature the teacher answers, and which answers you keep. The rest of this article is about that.

The mechanism: the student is only constrained on the problems you drew

One equation is worth pinning down. Distillation loss can be written roughly like this.

Symbol by symbol: $x$ is an input (a prompt), $q$ is the distribution you draw inputs from when building the distillation set, $p_T$ is the teacher's output probability, $p_S$ the student's, $\theta$ the student's parameters, and $\mathrm{KL}$ measures the gap between two probability distributions (KL divergence).

In plain words, the equation says: "make the teacher and student answer alike, but only on the problems $q$ picked." Flip that around and it says the equation demands nothing at all of the student on inputs $q$ never drew. Whatever the student says there is an accident that happened outside of training.

That is the starting point for data design. The teacher's intelligence sets the quality of $p_T$, but $q$ is ours to choose — and a student's weaknesses tend to be shaped exactly like the holes in $q$.

Why to skew toward hard problems — where the gradient lives

"More is better" is not the rule. With softmax and cross-entropy, the gradient of the loss with respect to the student's logits $z_S$ takes a strikingly simple form.

Here $i$ indexes a candidate (one token in the vocabulary), and $p_S^{(i)}$, $p_T^{(i)}$ are the probabilities the student and teacher assign to it.

So the size of the learning signal is literally the size of the disagreement. On examples where the student already outputs the teacher's probabilities, the difference is zero and the gradient vanishes. You called the teacher, you spun the GPUs, and not one parameter moved.

That is why it pays to deliberately skew distillation data toward where the student is currently failing. Three recipes are common in practice: (1) have the student attempt candidate prompts and forward only the ones it got wrong to the teacher; (2) sample the teacher $k$ times and discard problems it gets right every time (too easy) and wrong every time (beyond the teacher too), keeping the middle; (3) use failed problems as seeds and generate variants with different phrasings or numbers (Self-Instruct-style expansion).

"The same problem over and over" looks like volume but isn't

The dangerous thing about synthetic data is how easily you land in a state where the row count grows and the information doesn't. Sample the teacher at a low temperature and the same prompt returns nearly the same answer; you think you built 100,000 examples and the contents are really a few thousand phrasings repeated. The student memorizes those, training loss falls beautifully, and it collapses on slightly shifted inputs.

The fixes are unglamorous: drop one of each pair whose embeddings exceed a cosine-similarity threshold, strip n-gram reuse, and above all diversify the inputs before you try to diversify the outputs.

Correctness filtering — not copying the mistakes too

Teachers make mistakes. And distillation copies those mistakes into the student verbatim. Worse, the student also picks up the confident tone that came with them.

For tasks you can check — a matching final answer in math, a passing test suite in code — this is mechanically solvable. Sample the teacher $k$ times on the same problem and keep only the traces that reached the correct answer. That is rejection sampling, the skeleton STaR laid out. The crucial part is that what you keep is not the answer but the reasoning that got there — the answers were already in your original dataset.

The trap is that traces which reach the right answer through wrong reasoning slip through. On multiple choice, a guess is sometimes correct. Distillation works better on tasks with strict checks precisely because there are fewer such false positives. Summarization and advice have no such safety net, so you fall back on human review or a grader model.

Why DeepSeek-R1's distillation worked

DeepSeek-R1 (2025) reads as a direct implementation of everything above.

The reported pipeline: take R1, whose reasoning was developed through reinforcement learning, as the teacher, and build roughly 800k training examples from it — about 600k reasoning examples and about 200k non-reasoning ones (general question answering, writing, and so on). The reasoning half was built with the rejection sampling described above. Those 800k examples were then used to fine-tune small off-the-shelf Qwen and Llama models with ordinary supervised learning. No reinforcement learning was run on the student side.

The comparison reported alongside it matters: distilling from a large teacher beat applying reinforcement learning directly to the small model. RL pays no reward unless the model stumbles onto a correct trajectory by itself, and a small model almost never finds one on hard problems, so learning stalls. Distillation outsources that search to the teacher.

So what R1's distillation transported was less knowledge than the fruits of search: behind the 800k sits an inventory of "solution procedures that work," found by a teacher burning a great deal of GPU time. The model lineage itself is covered in The DeepSeek Family.

In code: the data design loop

The skeleton is a bit over ten lines. Swap in your own teacher call and checker and it applies to your task.

def build_distill_data(prompts, teacher, student, check, k=8, temp=0.8):
    data = []
    for p in prompts:
        if student_solves(student, p):      # already solved -> no gradient
            continue
        cands = [teacher(p, temperature=temp) for _ in range(k)]
        ok = [c for c in cands if check(p, c)]   # rejection sampling
        if not ok or len(ok) == k:          # all fail = too hard / all pass = too easy
            continue
        data.append((p, pick_shortest(ok)))  # avoid bloated traces
    return dedupe_by_embedding(data, threshold=0.95)

Discarding len(ok) == k is the point: a problem solved 8 times out of 8 carries almost no learning signal, so the same budget goes further on mid-difficulty items. pick_shortest has a reason too — a student trained on long traces imitates the token count at inference time as well, so your serving cost rises with it.

How this is used in practice

Who touches it, and when. ML engineers shrinking in-house models, or applied teams building domain-specific ones. The trigger is usually "the per-call price of the API doesn't work, we want to serve a 7B ourselves," and the first week or two goes into building the data generation pipeline, not training the model.

Parameters and tools you actually touch. On the generation side: temperature (0.7–1.0 is typical; too low gives duplicates, too high gives wrong answers), top_p, and $k$, the number of samples per prompt. On the filtering side: the checker script, the near-duplicate similarity threshold, and a length cap. Batch generation usually runs on vLLM or SGLang; deduplication is typically MinHash/SimHash or embeddings plus approximate nearest neighbors.

Pitfalls that turn into incidents.

1. Eval set contamination. Auto-expanding seed prompts sneaks paraphrases of benchmark problems into training data. Scores go up and ability doesn't — the hardest failure to notice. Check seeds against your eval sets before generating.
2. The teacher's terms of service. Using a commercial API's output to train a competing model is prohibited by many providers' terms. Technically possible and contractually allowed are different questions.
3. Distribution skew after filtering. Keeping only correct traces pulls the dataset toward problems the teacher is good at; its weak areas quietly disappear. Compare per-category counts before and after filtering.
4. Over-tightening the difficulty filter. Feed it only hard problems and the student forgets how to respond plainly to easy inputs. The 20% non-reasoning data in R1 is that insurance policy.
5. Building the set once. When the student changes, so does the set of problems it already solves. Data design is a loop: look at the student's failures, then choose the next $q$.

How it shows up in interviews. "Same teacher, but two distilled students differ a lot in quality. What differed?" The line of reasoning is to suspect $q$, not the loss function: coverage, duplication rate, difficulty distribution, and the filter's false-positive rate, in that order.

Summary

• Modern LLM distillation is sequence-level, so nearly all design freedom sits on the data side
• The loss aligns teacher and student only on inputs $q$ drew; holes in $q$ become the student's weaknesses
• Gradients scale with disagreement, so problems the student already solves teach it nothing
• Rejection sampling keeps only correct traces — mind the false positives and the skew the filter creates
• R1's distillation worked because the teacher stocked an inventory of correct trajectories the small model could never have found alone

Next we look at cases where distillation fails even with well-designed data: the capacity gap between teacher and student, and the contagion of overconfidence.