Evaluating Distilled Models — Is "Close to the Teacher" a Good Metric?

Score a distilled student by how often it agrees with its teacher and the students that faithfully reproduce the teacher's mistakes come out on top. What agreement actually guarantees, what breaks outside the training distribution, and how to test for contamination that arrives paraphrased through synthetic data.

The False Promise of Imitating Proprietary LLMs

Grading the student on how often they matched the teacher

Distillation copies the behaviour of a large teacher model into a smaller student. Since the training objective is "match the teacher," it feels natural to reach for the same yardstick at evaluation time: push a test set through both models and count how often the outputs agree. The number is cheap to produce and needs no gold labels at all.

But that is grading a student on how many answers matched the teacher's paper, not on the exam itself. If the teacher were perfect, agreement and ability would coincide. Real teacher models are not perfect. When the teacher gets a question wrong and the student gets it wrong the same way, agreement goes up. The metric is biased, by construction, in favour of students that faithfully reproduce the teacher's errors.

Worse, high-agreement models look like drop-in replacements. Surface properties — tone, phrasing, formatting, the habit of answering with confidence — transfer from very few samples, so the student reads like the teacher almost immediately. Factual accuracy and multi-step reasoning do not transfer nearly so cheaply. Everything hard about evaluating distilled models lives in that gap between looking close and being close.

What agreement does and does not tell you

Let's be precise. For a test set, write $\mathrm{Agr}$ for the fraction of items where teacher and student produce the same output, $a_T$ for the teacher's accuracy, and $a_S$ for the student's. The following always holds.

In words: high agreement bounds the gap in scores between teacher and student. It says nothing whatsoever about the level of either score.

The derivation is two lines. Whenever the student is right and the teacher is wrong, the two necessarily disagree, so $a_S - a_T \le P(\text{disagree}) = 1 - \mathrm{Agr}$. Swap their roles for the other direction.

As a practical guarantee this is weaker than it sounds. At 90% agreement, all you may conclude is that the student's accuracy lies within $\pm 10$ points of the teacher's. If the teacher scores 80, the student could be at 70 or at 90 — a band far too wide to decide a model swap on.

Decomposing the agreement makes the bias explicit.

$C$ is capability that transferred; $M$ is quirks and errors that transferred. A single agreement number adds them together, so it cannot distinguish a student that inherited the teacher's blind spots wholesale from one that reaches the same score with a different profile of strengths.

Put numbers on it. Teacher at 80, agreement at 90%. $C$ means "both correct," so it cannot exceed the teacher's 80 — which forces $M$ to be at least 10 points. If $M$ is exactly 10, then $C$ is 80 and the student lands somewhere between 80 and 90. If $M$ swells to 20, $C$ drops to 70 and so does the student. The same "90% agreement" describes both the strong student and the degraded one.

There is also a floor from chance agreement. On four-way multiple choice, two models answering at random still match a bit more than a fifth of the time, and on a task whose answers are dominated by one common label the floor is far higher than that. Raw agreement therefore starts from a number that has nothing to do with distillation having worked. If you use agreement diagnostically, subtract that baseline: $\kappa = (\mathrm{Agr} - \mathrm{Agr}_0) / (1 - \mathrm{Agr}_0)$ is the honest version of the quantity.

What breaks the moment you leave the distribution

Gudibande et al., in "The False Promise of Imitating Proprietary LLMs," measured this divergence. Models trained to imitate the outputs of a strong proprietary system came close to the target under human preference comparisons, while gaps remained on standard benchmarks probing factual accuracy and reasoning. Imitation buys the voice cheaply; it does not buy the capability. When a person picks between two outputs, the choice is pulled hard by register and confident phrasing, so copying the style raises the win rate on its own.