#temperature
3 articles
01
·Distillation & Compression·FREE·PAPER·8 min read
The Math of Distillation — Why Soft Answers Teach More
Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer.
02
·Distillation & Compression·★ MEMBER·PAPER·11 min read
Build Your Own Distillation — Growing a Small Model in 100 Lines
The distillation loss fits in twenty lines — and almost everyone who writes it trips on the same three things: the direction of the KL, the choice of reduction, and the missing T². We build the whole rig: freezing the teacher, the loss, the training loop, the teacher-free baseline, a temperature sweep, and four sanity checks that prove the implementation isn't quietly broken.
03
·Large Language Models·★ MEMBER·PAPER·9 min read
Knowledge Distillation from Scratch — Copying a Big Model into a Small One
No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.