#training
33 articles
01
·Distillation & Compression·★ MEMBER·PAPER·9 min read
On-Policy Distillation — Learning From What the Student Actually Writes
Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD.
02
·Distillation & Compression·★ MEMBER·PAPER·9 min read
On-Policy Distillation — Learning From What the Student Actually Writes
Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD.
03
·Distillation & Compression·★ MEMBER·PAPER·9 min read
A Field Guide to Distillation Recipes — logit, feature, attention, self
Every distillation recipe is an answer to one question: which part of the teacher should the student match? This guide lines up output (logit), intermediate feature (FitNet), attention, and self-distillation in one table and four equations, then turns the choice into a decision you make from constraints — is the teacher behind an API, do the layers line up, do the head counts match.
04
·Distillation & Compression·★ MEMBER·PAPER·9 min read
A Field Guide to Distillation Recipes — logit, feature, attention, self
Every distillation recipe is an answer to one question: which part of the teacher should the student match? This guide lines up output (logit), intermediate feature (FitNet), attention, and self-distillation in one table and four equations, then turns the choice into a decision you make from constraints — is the teacher behind an API, do the layers line up, do the head counts match.
05
·Distillation & Compression·FREE·PAPER·8 min read
The Math of Distillation — Why Soft Answers Teach More
Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer.
06
·Distillation & Compression·FREE·PAPER·8 min read
The Math of Distillation — Why Soft Answers Teach More
Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer.
07
·Distillation & Compression·★ MEMBER·PAPER·10 min read
Distilling Agents — How to Compress a Long Trajectory
When you distill an agent, the unit of learning changes from a single question-and-answer to an entire game. Tens of thousands of tokens come back with one bit of grading. This article works through how that gap is bridged — turn-level credit assignment, trajectory filtering and on-policy correction, and the inheritance of tool use — and maps the recent long-horizon agent literature onto those three axes.
08
·Distillation & Compression·★ MEMBER·PAPER·8 min read
When Distillation Fails — Capacity Gaps and Contagious Overconfidence
A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles.
09
·Distillation & Compression·★ MEMBER·PAPER·8 min read
When Distillation Fails — Capacity Gaps and Contagious Overconfidence
A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles.
10
·Distillation & Compression·★ MEMBER·PAPER·8 min read
Designing Distillation Data — Deciding What to Ask the Teacher
What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles.
11
·Distillation & Compression·★ MEMBER·PAPER·8 min read
Designing Distillation Data — Deciding What to Ask the Teacher
What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles.
12
·Distillation & Compression·★ MEMBER·PAPER·11 min read
Build Your Own Distillation — Growing a Small Model in 100 Lines
The distillation loss fits in twenty lines — and almost everyone who writes it trips on the same three things: the direction of the KL, the choice of reduction, and the missing T². We build the whole rig: freezing the teacher, the loss, the training loop, the teacher-free baseline, a temperature sweep, and four sanity checks that prove the implementation isn't quietly broken.
13
·Distillation & Compression·★ MEMBER·PAPER·11 min read
Build Your Own Distillation — Growing a Small Model in 100 Lines
The distillation loss fits in twenty lines — and almost everyone who writes it trips on the same three things: the direction of the KL, the choice of reduction, and the missing T². We build the whole rig: freezing the teacher, the loss, the training loop, the teacher-free baseline, a temperature sweep, and four sanity checks that prove the implementation isn't quietly broken.
14
·Training & Alignment·FREE·PAPER·11 min read
Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart
Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge.
15
·Training & Alignment·FREE·PAPER·11 min read
Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart
Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge.
16
·Large Language Models·★ MEMBER·PAPER·9 min read
Knowledge Distillation from Scratch — Copying a Big Model into a Small One
No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.
17
·Large Language Models·★ MEMBER·PAPER·9 min read
Knowledge Distillation from Scratch — Copying a Big Model into a Small One
No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.
18
·Deep Learning Basics·★ MEMBER·PAPER·12 min read
Hyperparameter Search — Hunches, Grids, and Bayesian Optimization
Gradients tell you nothing about the learning rate, so you have to go looking. Why grid search is weak, why search spaces should be carved on a log scale, what a Bayesian acquisition function is actually counting, and why early stopping beats a cleverer search algorithm — with Optuna code and the traps that bite in production.
19
·Deep Learning Basics·★ MEMBER·PAPER·12 min read
Hyperparameter Search — Hunches, Grids, and Bayesian Optimization
Gradients tell you nothing about the learning rate, so you have to go looking. Why grid search is weak, why search spaces should be carved on a log scale, what a Bayesian acquisition function is actually counting, and why early stopping beats a cleverer search algorithm — with Optuna code and the traps that bite in production.
20
·Inference & Serving·★ MEMBER·PAPER·11 min read
Surviving GPU Out-of-Memory — Every Cause, Every Fix
`CUDA out of memory` reports only the allocation that happened to fail last, which is almost never the culprit. We count what actually occupies VRAM in five buckets, derive the sixteen-bytes-per-parameter fixed cost of training, and work through the fixes in order of least damage: gradient checkpointing, optimizer compression, offloading, KV cache limits, and fragmentation.
21
·Training & Alignment·★ MEMBER·PAPER·13 min read
DPO and What Came After — The Lineage That Simplified RLHF
Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have.
22
·Training & Alignment·★ MEMBER·PAPER·9 min read
Continual Learning and Catastrophic Forgetting — Why Models Can't Just Keep Learning
Why a model that learns something new tends to lose what it already knew, explained from a single fact: weights are a shared resource. Walks through EWC, replay, and swappable LoRA adapters with equations and interactive figures, then explains why production teams still just mix the old data back in and retrain.
23
·Deep Learning Basics·★ MEMBER·PAPER·15 min read
Weight Initialization and Regularization — What Lets Training Start, and What Keeps It Going
The same architecture will train or refuse to train depending on nothing more than how large the random numbers in its weights were. Starting from how variance propagates, this piece derives Xavier and He initialization, then turns to weight decay and dropout as the conditions that keep training going — with interactive figures, PyTorch code and the mistakes that actually cost people days.
24
·Deep Learning Basics·★ MEMBER·PAPER·15 min read
Weight Initialization and Regularization — What Lets Training Start, and What Keeps It Going
The same architecture will train or refuse to train depending on nothing more than how large the random numbers in its weights were. Starting from how variance propagates, this piece derives Xavier and He initialization, then turns to weight decay and dropout as the conditions that keep training going — with interactive figures, PyTorch code and the mistakes that actually cost people days.
25
·Deep Learning Basics·FREE·PAPER·13 min read
Activation Functions from Scratch — Why Nonlinearity Is Non-Negotiable
Without an activation function, a hundred stacked layers can do exactly what one layer does. Starting from that one-line proof, this article traces why sigmoid was abandoned, why ReLU won, and why today's LLMs settled on SiLU and SwiGLU — with an interactive plot where you can drag the input and watch the slope.
26
·Deep Learning Basics·FREE·PAPER·13 min read
Activation Functions from Scratch — Why Nonlinearity Is Non-Negotiable
Without an activation function, a hundred stacked layers can do exactly what one layer does. Starting from that one-line proof, this article traces why sigmoid was abandoned, why ReLU won, and why today's LLMs settled on SiLU and SwiGLU — with an interactive plot where you can drag the input and watch the slope.
27
·Deep Learning Basics·★ MEMBER·PAPER·10 min read
A History of Normalization Layers — From BatchNorm to RMSNorm
The layer that made deep learning actually deep, explained from zero. The internal-covariate-shift controversy behind BatchNorm, why LayerNorm threw away the batch axis, and why every modern LLM converged on RMSNorm.
28
·Deep Learning Basics·★ MEMBER·PAPER·10 min read
A History of Normalization Layers — From BatchNorm to RMSNorm
The layer that made deep learning actually deep, explained from zero. The internal-covariate-shift controversy behind BatchNorm, why LayerNorm threw away the batch axis, and why every modern LLM converged on RMSNorm.
29
·Training & Alignment·★ MEMBER·PAPER·10 min read
Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking
Halve the bits and training gets faster — right up until your gradients quietly turn into zeros or infs. We start from the two ways it breaks, work out what loss scaling actually does, why bf16 made it unnecessary, and what came back with fp8. Ends with the PyTorch AMP knobs and the clipping-order mistake that fails silently.
30
·Training & Alignment·★ MEMBER·PAPER·10 min read
Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking
Halve the bits and training gets faster — right up until your gradients quietly turn into zeros or infs. We start from the two ways it breaks, work out what loss scaling actually does, why bf16 made it unnecessary, and what came back with fp8. Ends with the PyTorch AMP knobs and the clipping-order mistake that fails silently.
31
·Training & Alignment·★ MEMBER·PAPER·10 min read
Learning Rate Schedules — Why Warmup and Why Cosine
The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs.
32
·Training & Alignment·★ MEMBER·PAPER·10 min read
Learning Rate Schedules — Why Warmup and Why Cosine
The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs.
33
·Parallel & Distributed·★ MEMBER·10 min read
Distributed Training from Scratch — Data Parallel, Model Parallel, and When Communication Becomes the Bottleneck
Why one machine is not enough, counted out in bytes; data parallelism and all-reduce; what ZeRO and FSDP actually shard; tensor and pipeline parallelism. Then the ratio of computation to communication that tells you where scaling stops paying — and gradient accumulation, NCCL settings and how to diagnose a hang.