Alignment, Explained — From RLHF to Constitutional AI

What exactly is being aligned with what? This piece separates the gap into three layers — objective, specification, and proxy — shows that RLHF, DPO and Constitutional AI all solve the same skeleton of "reward minus a penalty for drifting from the reference model," explains why pushing that proxy hard always produces reward hacking, and works through the trade between refusing and being useful, down to the implementation traps.

Training language models to follow instructions with human feedback

What is alignment actually aligning?

You ask the genie to make you rich and you inherit money from a relative who just died. Genie stories are unsettling not because the genie is malicious, but because the wish you said out loud differs from the thing you wanted. The more literally and the more forcefully your wish is granted, the more that gap turns into an accident.

Language models are the same shape. Pretraining maximizes "predict the next token of text on the web," not "follow the user's intent, be useful, and cause no harm." The InstructGPT paper we read in Instruction Tuning and RLHF calls this state misaligned, and names three things to align to: helpful, honest, harmless.

One term deserves care up front. Alignment is not the work of making a model smarter. What it touches is not capability (what the model can do) but propensity (what it tends to do).

The gap appears in three layers

"Misalignment" is one word for three different failures, and conversations go in circles when they get mixed.

1. The objective gap. Next-token prediction is simply a different objective from following instructions. Supervised fine-tuning (SFT) on demonstrations, plus learning from preferences, closes this one.

2. The specification gap. Nobody can write down "useful" and "harmless" in a form everyone agrees with. Even human labelers disagree with each other. So the spec can only be conveyed by example (preference data) or by principle (the constitution, below). It cannot be written directly as a loss function — that is what defines this layer.

3. The proxy gap. Inside the training loop, what the model is actually pushing up is not human satisfaction but the number emitted by a reward model, which is only a finite-data approximation of people. This layer is where reward hacking breeds.

Intuition: push the proxy hard enough and it detaches from the real thing

Goodhart's law puts it well: the moment a measure becomes a target, it stops being a good measure. Test scores aren't a bad way to gauge learning, but maximize scores alone and what grows is memorization and reflexes for past exam questions.

The reward model is exactly that test. Early on, raising the proxy raises the real thing too. The trouble starts when optimization keeps pushing: the reward model's score still climbs while human judgments plateau and then decline. This turnaround has been measured as reward model overoptimization, and smaller reward models turn around sooner.

The skeleton: every method solves "reward minus a penalty for drifting"

RLHF, DPO, Constitutional AI — the names keep multiplying, but the problem being solved is nearly identical.

Read aloud: "make the average reward as large as you can, but pay a fine of $\beta$ for every unit you drift away from the model you started with." Here $x$ is the prompt, $y$ the response, $\pi_\theta$ the model being trained, $\pi_{\mathrm{ref}}$ the reference model frozen before training, $r$ the reward, $D_{\mathrm{KL}}$ a measure of how far apart two distributions are, and $\beta$ the knob for how adventurous you are allowed to be.

Put in words, equation (1) is a single ledger with two columns: the model gains by producing answers people rate highly, and loses by producing answers far from the one it started as. $\beta$ is the exchange rate between the two columns — turn it up and the standing order becomes "don't drift from who you were"; turn it down and it becomes "do whatever scores points."