distill

Definition

To transfer the behaviour of a large teacher model into a smaller student. Instead of training the student on hard labels alone, you fit it to the teacher's full output distribution, or to its intermediate representations, which carries far more information per example than a one-hot target. Distinguish it from pruning and quantization: those shrink the original network, while distillation trains a different, smaller one.

Explainers using this term

• Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-ImprovementDoes On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement