Learn › AI

Training & Alignment

Pretraining, SFT, RLHF/DPO, fine-tuning

01 ·Training & Alignment·FREE·PAPER·9 min read Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing. 02 ·Training & Alignment·FREE·PAPER·10 min read Learning Rate Schedules — Why Warmup and Why Cosine The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs. 03 ·Training & Alignment·★ MEMBER·PAPER·10 min read Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking Halve the bits and training gets faster — right up until your gradients quietly turn into zeros or infs. We start from the two ways it breaks, work out what loss scaling actually does, why bf16 made it unnecessary, and what came back with fp8. Ends with the PyTorch AMP knobs and the clipping-order mistake that fails silently. 04 ·Training & Alignment·FREE·PAPER·10 min read LoRA vs. Full Fine-Tuning — What You Give Up, What You Get LoRA squeezes a model's weight update into two thin matrices, ΔW = BA. This article starts from zero: why that is enough, what you actually trade away against full fine-tuning, how to pick the rank, and where merging quietly breaks. 05 ·Training & Alignment·★ MEMBER·PAPER·11 min read Building a Dataset in Practice — Collect, Clean, Blend Most of the work in building a model is really the work of building its dataset. From designing the population you sample from, to calibrating filter thresholds, measuring leakage into your test set, converting mixture ratios into epochs, using synthetic data where it belongs, and writing annotation guidelines that actually hold — explained from zero, with the formulas and the procedures. 06 ·Training & Alignment·★ MEMBER·PAPER·13 min read DPO and What Came After — The Lineage That Simplified RLHF Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have. 07 ·Training & Alignment·★ MEMBER·PAPER·9 min read Continual Learning and Catastrophic Forgetting — Why Models Can't Just Keep Learning Why a model that learns something new tends to lose what it already knew, explained from a single fact: weights are a shared resource. Walks through EWC, replay, and swappable LoRA adapters with equations and interactive figures, then explains why production teams still just mix the old data back in and retrain. 08 ·Training & Alignment·FREE·PAPER·11 min read Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge. 09 ·Training & Alignment·★ MEMBER·11 min read Versioning Data and Models — An Experiment You Cannot Reproduce Never Happened If you cannot get "92% accuracy" back six months later, it was an anecdote, not an experiment. We build up content-addressing (naming things by their contents), lineage (the graph from inputs to outputs), and the three levels of "how much is worth pinning down" — from the hash math to a manifest you can write today to the mistakes that quietly ruin a year of runs. 10 ·Training & Alignment·★ MEMBER·PAPER·11 min read Paper Deep-Dive: ABSeeker — Training Long-Horizon Search Agents by Grading Each Step Backward from the Answer When an agent chains dozens of web searches, the only reward is whether the final answer was right — so every good move and every bad move along the way gets graded identically. We walk through ABC (Answer-Backtracked Credit Assignment), which recovers clues by working backward from the answer and scores every step against them, and ABSeeker, the 4B model trained with it — all from the paper itself. 11 ·Training & Alignment·★ MEMBER·PAPER·10 min read PAWBench Explained — Can Video Generators Get the Odds Right, Not Just the Physics? If a video generator is a world model, it owes you more than one plausible rollout — it owes you the right distribution over futures. PAWBench measures that probabilistic alignment across 50 scenarios and 11 systems, and finds that no model gets all the requirements at once. 12 ·Training & Alignment·★ MEMBER·PAPER·11 min read Paper Walkthrough: PaperGym — Turning One Paper Into a Graded Training Environment for Research Plans A research plan has no answer key, so reinforcement learning has no environment. This paper splits each paper into four drawers and builds the question and the grading criteria from different drawers, producing 20,000 training environments where paraphrasing the prompt no longer earns reward. 13 ·Training & Alignment·★ MEMBER·PAPER·12 min read Paper Walkthrough: StudentSim — Training a Simulator That Is Actually *That* Student How to build the practice students an AI tutor needs, by training on real learner records. From the two-axis definition of behavioral fidelity and guidance responsiveness, through the pooled-then-specialized pipeline, to using the simulator as a reward model. 14 ·Training & Alignment·★ MEMBER·PAPER·13 min read Paper Walkthrough: It Takes Two to Match — Co-Evolving Both Sides of Retrieval with RL A paper that rebuilds the first stage of search by having two LLMs write keywords — one for queries, one for items — and training them in alternation until their vocabularies meet. Explained from scratch. 15 ·Training & Alignment·★ MEMBER·PAPER·14 min read Paper Explained: DataFlex-RL — Did 11 RLVR Data Policies Beat Uniform Sampling? Re-measured Over 12 Seeds "Train only on the good rollouts and the model gets stronger" is fast becoming RLVR folk wisdom. DataFlex-RL (arXiv:2609.06107) compares 13 selection, reweighting and mixture-adaptation configurations under one GRPO recipe and 12 seeds, and reports that every difference from uniform sampling has a 95% interval that crosses zero. We walk through the paper, including the finding that the choice of evaluation benchmarks reshuffles the winners. 16 ·Training & Alignment·★ MEMBER·PAPER·10 min read Paper Walkthrough: SpeakerMem-R1 — A Two-Track Memory That Remembers Who Said What, About Whom, and When A from-scratch look at SpeakerMem-R1, a long-term memory for group conversations. It keeps verbatim messages alongside person- and group-level states, and trains a small note-taking model (Qwen2.5-3B) with reinforcement learning that scores memory owner by owner. 17 ·Training & Alignment·★ MEMBER·PAPER·8 min read Paper walkthrough: WanPE — turning a video prompt into a director's shooting plan WanPE rewrites a one-line request into a shot-by-shot plan covering camera, lighting and sound. We explain, from zero, why it builds training pairs backwards from real videos and how SC-GRPO penalizes dropped requirements. 18 ·Training & Alignment·★ MEMBER·PAPER·7 min read Paper Explained: When Wording Changes an AI Reviewer's Score — RobustReview and SciCore A from-scratch walkthrough of a paper that defines 'rhetorical robustness' for AI reviewers whose scores shift when the same science is reworded. We cover 'false robustness', where stability alone fools you, and SciCore, which averages a full-manuscript judgment with a judgment of an extracted science core. 19 ·Training & Alignment·★ MEMBER·PAPER·7 min read Paper Deep Dive: Transformers Stop Thinking Too Early — How a One-Layer Rank-8 LoRA Stretches 3-Line Reference Following to 50 Pretrained LLMs follow in-context variable references for only two or three lines. Add a rank-8 LoRA (about 65K parameters) at one early layer with every weight frozen, and Qwen3-8B goes from 15.5% to 99% on 24-line chains. We walk through the mid-layer 'relay' behind it and the sharp limit on where the fix can go. 20 ·Training & Alignment·FREE·PAPER·19 min read QLoRA — Fine-Tuning a 65B Model on One GPU A 65-billion-parameter model in 16-bit weights is 130 GB before you have trained anything. QLoRA gets that onto a single 48 GB card by storing the frozen weights in a 4-bit format shaped like the weights themselves, quantizing the quantization constants, and paging the optimizer to CPU — while gradients still flow through those 4-bit weights into a small 16-bit adapter. 21 ·Training & Alignment·FREE·PAPER·17 min read LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time. 22 ·Training & Alignment·FREE·PAPER·18 min read Adam from Scratch — Two Moving Averages and the Correction Nobody Explains Adam is the default optimizer in almost every deep-learning codebase, and almost nobody can say what its four lines do. This article assembles it from the two problems it solves — the zigzag of plain SGD and the mismatched gradient scales between layers — then takes the bias correction apart with numbers: without it, Adam's very first step is 3.2× too large and the error peaks at 6.6×. Ends on the traps: weight decay belongs in AdamW, not Adam, and the epsilon sits outside the square root for a reason.