Training & Alignment
Pretraining, SFT, RLHF/DPO, fine-tuning
01
·Training & Alignment·★ MEMBER·PAPER·9 min read
Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders
A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing.
02
·Training & Alignment·★ MEMBER·PAPER·10 min read
Learning Rate Schedules — Why Warmup and Why Cosine
The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs.
03
·Training & Alignment·★ MEMBER·PAPER·10 min read
Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking
Halve the bits and training gets faster — right up until your gradients quietly turn into zeros or infs. We start from the two ways it breaks, work out what loss scaling actually does, why bf16 made it unnecessary, and what came back with fp8. Ends with the PyTorch AMP knobs and the clipping-order mistake that fails silently.
04
·Training & Alignment·★ MEMBER·PAPER·11 min read
Building a Dataset in Practice — Collect, Clean, Blend
Most of the work in building a model is really the work of building its dataset. From designing the population you sample from, to calibrating filter thresholds, measuring leakage into your test set, converting mixture ratios into epochs, using synthetic data where it belongs, and writing annotation guidelines that actually hold — explained from zero, with the formulas and the procedures.
05
·Training & Alignment·★ MEMBER·PAPER·13 min read
DPO and What Came After — The Lineage That Simplified RLHF
Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have.
06
·Training & Alignment·★ MEMBER·PAPER·9 min read
Continual Learning and Catastrophic Forgetting — Why Models Can't Just Keep Learning
Why a model that learns something new tends to lose what it already knew, explained from a single fact: weights are a shared resource. Walks through EWC, replay, and swappable LoRA adapters with equations and interactive figures, then explains why production teams still just mix the old data back in and retrain.
07
·Training & Alignment·FREE·PAPER·11 min read
Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart
Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge.
08
·Training & Alignment·★ MEMBER·10 min read
Versioning Data and Models — An Experiment You Cannot Reproduce Never Happened
If you cannot get "92% accuracy" back six months later, it was an anecdote, not an experiment. We build up content-addressing (naming things by their contents), lineage (the graph from inputs to outputs), and the three levels of "how much is worth pinning down" — from the hash math to a manifest you can write today to the mistakes that quietly ruin a year of runs.
09
·Training & Alignment·★ MEMBER·PAPER·11 min read
Paper Deep-Dive: ABSeeker — Training Long-Horizon Search Agents by Grading Each Step Backward from the Answer
When an agent chains dozens of web searches, the only reward is whether the final answer was right — so every good move and every bad move along the way gets graded identically. We walk through ABC (Answer-Backtracked Credit Assignment), which recovers clues by working backward from the answer and scores every step against them, and ABSeeker, the 4B model trained with it — all from the paper itself.
10
·Training & Alignment·★ MEMBER·PAPER·10 min read
PAWBench Explained — Can Video Generators Get the Odds Right, Not Just the Physics?
If a video generator is a world model, it owes you more than one plausible rollout — it owes you the right distribution over futures. PAWBench measures that probabilistic alignment across 50 scenarios and 11 systems, and finds that no model gets all the requirements at once.
11
·Training & Alignment·★ MEMBER·PAPER·10 min read
Paper Walkthrough: PaperGym — Turning One Paper Into a Graded Training Environment for Research Plans
A research plan has no answer key, so reinforcement learning has no environment. This paper splits each paper into four drawers and builds the question and the grading criteria from different drawers, producing 20,000 training environments where paraphrasing the prompt no longer earns reward.
12
·Training & Alignment·★ MEMBER·PAPER·12 min read
Paper Walkthrough: StudentSim — Training a Simulator That Is Actually *That* Student
How to build the practice students an AI tutor needs, by training on real learner records. From the two-axis definition of behavioral fidelity and guidance responsiveness, through the pooled-then-specialized pipeline, to using the simulator as a reward model.
13
·Training & Alignment·★ MEMBER·PAPER·13 min read
Paper Walkthrough: It Takes Two to Match — Co-Evolving Both Sides of Retrieval with RL
A paper that rebuilds the first stage of search by having two LLMs write keywords — one for queries, one for items — and training them in alternation until their vocabularies meet. Explained from scratch.