#rlhf
9 articles
01
·Evaluation & Judging·★ MEMBER·PAPER·10 min read
Reward Hacking — Whatever You Measure Is Where It Breaks
The moment you pick a metric, that metric starts to rot. This piece explains why Goodhart's law is statistically unavoidable, walks through real failures from boat races that spin in circles to RLHF verbosity, sycophancy and hardcoded unit tests, and covers how to detect the gap between optimization pressure and true performance.
02
·Paper Deep-Dives·★ MEMBER·PAPER·12 min read
Paper Walkthrough — J-Zero: Growing the Challenger, the Solver, and the Judge Together from Zero Data
A model that writes its own problems, solves them, and grades them — with no external data and no human labels. J-Zero's twist is that the grader learns too, which is what lets it keep improving past the two-iteration wall where prior methods stall.
03
·★ MEMBER·PAPER·9 min read
Paper Walkthrough: On-Policy Self-Distillation in Diffusion Models — Turning Reward into a Target You Can Aim At
When you train an image generator with rewards, a score on the finished picture never tells the model how to change its intermediate denoising predictions. DiffusionOPSD builds explicit targets from reward gradients and fits them using an EMA copy of the model itself. A ground-up explanation, strictly within what the abstract states.
04
·Training & Alignment·★ MEMBER·PAPER·13 min read
DPO and What Came After — The Lineage That Simplified RLHF
Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have.
05
·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Annotations as Rollouts — Dropping the Ground Truth Into the Group as a Ninth Answer
RL post-training for video MLLMs starves because sampled rollouts almost never contain the right answer. OraRL adds the annotation itself to the group as an extra rollout, then repairs the advantage inversion that naive mixing causes.
06
·Large Language Models·★ MEMBER·PAPER·8 min read
Alignment, Explained — From RLHF to Constitutional AI
What exactly is being aligned with what? This piece separates the gap into three layers — objective, specification, and proxy — shows that RLHF, DPO and Constitutional AI all solve the same skeleton of "reward minus a penalty for drifting from the reference model," explains why pushing that proxy hard always produces reward hacking, and works through the trade between refusing and being useful, down to the implementation traps.
07
·Large Language Models·★ MEMBER·PAPER·11 min read
Paper Walkthrough: SA-MRPO — Stop Studying the Subject You've Already Aced
When you train with several reward objectives at once, gradient budget keeps flowing to objectives that are already solved. SA-MRPO (arXiv:2608.16072) measures how saturated each objective is and discounts its weight accordingly, redirecting optimization toward whatever headroom remains. A walkthrough grounded strictly in the paper.
08
·Information Theory·★ MEMBER·PAPER·9 min read
KL Divergence From Scratch — Measuring the Gap Between Two Distributions
KL divergence measures the gap between two probability distributions. We build it up from a compression metaphor to the definition, its famous asymmetry, and a numpy implementation — then watch it at work as the regularizer in VAEs and the leash in RLHF.
09
·Training & Alignment·★ MEMBER·PAPER·9 min read
Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders
A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing.