#knowledge-distillation

1 articles

01 ·Distillation & Compression·FREE·PAPER·9 min read A Field Guide to Distillation Recipes — logit, feature, attention, self Every distillation recipe is an answer to one question: which part of the teacher should the student match? This guide lines up output (logit), intermediate feature (FitNet), attention, and self-distillation in one table and four equations, then turns the choice into a decision you make from constraints — is the teacher behind an API, do the layers line up, do the head counts match.