#kl-divergence
7 articles
01
·Distillation & Compression·★ MEMBER·PAPER·9 min read
On-Policy Distillation — Learning From What the Student Actually Writes
Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD.
02
·Distillation & Compression·FREE·PAPER·8 min read
The Math of Distillation — Why Soft Answers Teach More
Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer.
03
·Distillation & Compression·★ MEMBER·PAPER·8 min read
When Distillation Fails — Capacity Gaps and Contagious Overconfidence
A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles.
04
·Distillation & Compression·★ MEMBER·PAPER·11 min read
Build Your Own Distillation — Growing a Small Model in 100 Lines
The distillation loss fits in twenty lines — and almost everyone who writes it trips on the same three things: the direction of the KL, the choice of reduction, and the missing T². We build the whole rig: freezing the teacher, the loss, the training loop, the teacher-free baseline, a temperature sweep, and four sanity checks that prove the implementation isn't quietly broken.
05
·Large Language Models·★ MEMBER·PAPER·9 min read
Knowledge Distillation from Scratch — Copying a Big Model into a Small One
No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.
06
·Information Theory·★ MEMBER·PAPER·9 min read
KL Divergence From Scratch — Measuring the Gap Between Two Distributions
KL divergence measures the gap between two probability distributions. We build it up from a compression metaphor to the definition, its famous asymmetry, and a numpy implementation — then watch it at work as the regularizer in VAEs and the leash in RLHF.
07
·Information Theory·★ MEMBER·9 min read
Information Theory and AI — Where Cross-Entropy Loss Came From
Information is surprise, entropy is average surprise, KL divergence is the gap between two distributions. Stack those three and the loss function you already use for classification falls out as the only reasonable answer — including why it involves a logarithm, and what perplexity is really counting.