hints

Definition

In distillation, guidance from the teacher beyond its final output — typically the activations of an intermediate layer, matched by an auxiliary loss on a chosen student layer. Hints carry more information than labels or output logits and can rescue students too deep to train directly, but they force design choices: which layers to pair, and how to reconcile differing widths.