fitnets

Definition

An early knowledge-distillation method that trains a thin, deep student by matching not only the teacher's output but also an intermediate layer — the hint — through a regressor that bridges the width mismatch. That intermediate supervision is what makes narrow deep students trainable at all. It is the standard citation for feature-based distillation, as opposed to matching soft output logits alone.