distillation

Definition

Training a smaller student model to reproduce a larger teacher's behavior, using the teacher's output distribution or internal representations as the training signal rather than hard labels alone. Soft targets carry more information than one-hot labels, so students often beat same-size models trained from scratch. Unlike quantization or pruning, distillation produces a genuinely new model rather than compressing an existing one.

Explainers using this term

• Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of StepsRethinking On-Policy Distillation of Large Language Models II: One Training Example
• Paper Walkthrough: SecOPD — Grading One Token at a Time to Cut Adaptive Prompt Injection by an Order of MagnitudeSecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation
• Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-ImprovementDoes On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement