Distillation & Compression
Moving capability from a large model into a small one — losses, data design and failure modes
01
·Distillation & Compression·FREE·PAPER·8 min read
The Math of Distillation — Why Soft Answers Teach More
Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer.
02
·Distillation & Compression·★ MEMBER·PAPER·9 min read
A Field Guide to Distillation Recipes — logit, feature, attention, self
Every distillation recipe is an answer to one question: which part of the teacher should the student match? This guide lines up output (logit), intermediate feature (FitNet), attention, and self-distillation in one table and four equations, then turns the choice into a decision you make from constraints — is the teacher behind an API, do the layers line up, do the head counts match.
03
·Distillation & Compression·★ MEMBER·PAPER·8 min read
Designing Distillation Data — Deciding What to Ask the Teacher
What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles.
04
·Distillation & Compression·★ MEMBER·PAPER·8 min read
When Distillation Fails — Capacity Gaps and Contagious Overconfidence
A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles.
05
·Distillation & Compression·★ MEMBER·PAPER·9 min read
On-Policy Distillation — Learning From What the Student Actually Writes
Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD.
06
·Distillation & Compression·★ MEMBER·PAPER·10 min read
Distillation vs. Quantization vs. Pruning — Three Roads to a Smaller Model
There are three roads to a smaller model: coarsen the number grid (quantization), remove weights outright (pruning), or rebuild the thing at a smaller size (distillation). A head-to-head comparison on compression ratio, accuracy, and implementation cost — and why, when you stack them, training-based methods go first and quantization goes last.
07
·Distillation & Compression·★ MEMBER·PAPER·10 min read
Distilling Agents — How to Compress a Long Trajectory
When you distill an agent, the unit of learning changes from a single question-and-answer to an entire game. Tens of thousands of tokens come back with one bit of grading. This article works through how that gap is bridged — turn-level credit assignment, trajectory filtering and on-policy correction, and the inheritance of tool use — and maps the recent long-horizon agent literature onto those three axes.
08
·Distillation & Compression·★ MEMBER·PAPER·9 min read
Evaluating Distilled Models — Is "Close to the Teacher" a Good Metric?
Score a distilled student by how often it agrees with its teacher and the students that faithfully reproduce the teacher's mistakes come out on top. What agreement actually guarantees, what breaks outside the training distribution, and how to test for contamination that arrives paraphrased through synthetic data.
09
·Distillation & Compression·★ MEMBER·PAPER·11 min read
Build Your Own Distillation — Growing a Small Model in 100 Lines
The distillation loss fits in twenty lines — and almost everyone who writes it trips on the same three things: the direction of the KL, the choice of reduction, and the missing T². We build the whole rig: freezing the teacher, the loss, the training loop, the teacher-free baseline, a temperature sweep, and four sanity checks that prove the implementation isn't quietly broken.