distilled

Definition

Describing a model whose knowledge came from a larger one via distillation. The student is trained against the teacher's soft output distribution, and sometimes its intermediate activations, which carries more signal per example than hard labels — the teacher's relative confidence across wrong answers is itself information. Distinct from quantization and pruning, which shrink an existing model's numerics or weights; distillation retrains a new one under a teacher's supervision.