#distribution-shift

1 articles

01 ·Distillation & Compression·★ MEMBER·PAPER·8 min read When Distillation Fails — Capacity Gaps and Contagious Overconfidence A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles.