distillation

Definition

Training a smaller student model to reproduce a larger teacher's behavior, using the teacher's output distribution or internal representations as the training signal rather than hard labels alone. Soft targets carry more information than one-hot labels, so students often beat same-size models trained from scratch. Unlike quantization or pruning, distillation produces a genuinely new model rather than compressing an existing one.

Explainers using this term

• Paper Explained: Beyond Teacher Assignment — Why Choosing the Right Teacher Isn't Enough When You Distill Many Specialists into One ModelBeyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
• Paper Explained: When EOS Tokens Disagree — Why On-Policy Distillation Students Forget How to StopWhen EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation
• Paper Explained: Negative Self-Distillation — Learning to Reason by Steering Away From Your Worst SelfNegative Self-Distillation: Learning to Reason by Avoiding Flaws