#alignment
6 articles
01
·Agents·★ MEMBER·PAPER·10 min read
Paper Walkthrough: SecOPD — Grading One Token at a Time to Cut Adaptive Prompt Injection by an Order of Magnitude
Defensively fine-tuned LLMs still fall to adaptive prompt injections at close to 100%. The culprit is training that hands the whole response a single score. SecOPD grades every token using a teacher that never saw the injection — explained from first principles.
02
·Evaluation & Judging·★ MEMBER·PAPER·10 min read
Reward Hacking — Whatever You Measure Is Where It Breaks
The moment you pick a metric, that metric starts to rot. This piece explains why Goodhart's law is statistically unavoidable, walks through real failures from boat races that spin in circles to RLHF verbosity, sycophancy and hardcoded unit tests, and covers how to detect the gap between optimization pressure and true performance.
03
·Training & Alignment·★ MEMBER·PAPER·13 min read
DPO and What Came After — The Lineage That Simplified RLHF
Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have.
04
·Large Language Models·★ MEMBER·PAPER·8 min read
Alignment, Explained — From RLHF to Constitutional AI
What exactly is being aligned with what? This piece separates the gap into three layers — objective, specification, and proxy — shows that RLHF, DPO and Constitutional AI all solve the same skeleton of "reward minus a penalty for drifting from the reference model," explains why pushing that proxy hard always produces reward hacking, and works through the trade between refusing and being useful, down to the implementation traps.
05
·Training & Alignment·★ MEMBER·PAPER·9 min read
Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders
A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing.
06
·Training & Alignment·★ MEMBER·PAPER·9 min read
Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders
A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing.