DPO and What Came After — The Lineage That Simplified RLHF

Derives DPO one line at a time, starting from the closed-form solution to KL-constrained reward maximization, to show why no separate reward model is needed. Then organizes IPO (which explains DPO's overfitting mathematically), KTO (which drops the pairing requirement), and GRPO (which drops the value model and goes back online) by what each one deleted — and gives a rule for choosing based on the shape of the data you actually have.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

The Detour Called a Reward Model

Say you want to get better at cooking. The obvious approach is to ask the people eating: "A or B — which was better?" and let the answer shape the next dish.

RLHF (reinforcement learning from human feedback) inserts a step in between. First you collect a pile of "A beats B" comparisons, and you use them to train a judge: a neural network that takes a dish and returns a score. That's the reward model. Then you use reinforcement learning to push the cook (the language model) toward dishes the judge scores highly. That three-stage pipeline is the InstructGPT recipe covered in Instruction Tuning and RLHF.

The reason for the detour is clear enough. RL runs "try, get scored, adjust" tens of thousands of times, and you cannot ask a human on every one of those steps. So you need a stand-in.

The cost is just as clear. Four models sit on the GPU at once: the policy being trained, a frozen reference model, the reward model, and the value model PPO needs. On top of that, RL is finicky. All of this to express "we'd like the model to match human preferences."

DPO (Direct Preference Optimization), from 2023, landed a clean hit here. You don't need the judge. More precisely: the judge is already written into the policy itself, so there is no reason to stand up a separate network for it.

The Intuition: The Scorecard Is Already Inside the Policy

The core claim fits on one line. Fix a scoring rule (a reward function) and the cook optimized for it is uniquely determined — and the reverse holds too. Watch how a cook behaves and you can work backward to the scoring rule they were optimized against. If that's true, there's no need to learn the scoring rule first. Apply the comparison data straight to the cook.

One caveat: working backward requires a reference point. If a cook salts heavily, you can't tell in isolation whether that's because salt scored well or because it was always their habit. Only by comparing against the pre-training version can you read off the shift that training pushed. That's why the reference model stays with DPO to the end. What disappears is the reward model and the value model — not the reference.

The Mechanism: Three Lines That Delete the Reward Model

We start from the problem RLHF is actually trying to solve.

Here $x$ is the input (prompt), $y$ the output (response), $\pi_\theta$ the policy being trained, $\pi_{\mathrm{ref}}$ the reference model frozen before training, $r(x,y)$ the reward, and $D_{\mathrm{KL}}$ the KL divergence measuring the gap between two distributions. Said in words, the whole line reads: "score as high as you can on average, but hand back $\beta$ times whatever distance you put between yourself and the model you started from." $\beta$ is the dial for how much wandering is allowed.

This is where the road forks. The old approach solved this approximately, with PPO. But a problem of this shape has an exact solution in closed form.

Unpacked in words, the optimal policy is just "take how likely the original model was to say it, multiply by the exponential of the reward, and divide so everything sums to one." $Z(x)$ is the normalizing constant (the partition function) used for that division, one value per $x$. High-reward responses get lifted by the exponential — but a response the original model almost never produces has a tiny $\pi_{\mathrm{ref}}$, so it won't suddenly appear. That is exactly what the KL penalty is.

Now solve that equation for $r$. Take logs and rearrange; that's all.

This is the decisive line. The reward is now written as a log ratio between the policy and the reference model. It is a sentence which says: a response's reward is nothing more than "how many times more likely the trained model is to say this than the original model was," taken in logs and scaled by $\beta$ — with $\log Z(x)$ an offset that lands on every response for a given prompt equally. You don't need to hold a reward function separately — the policy doubles as one. The paper's subtitle, "Your Language Model Is Secretly a Reward Model," is pointing at this equation.

What's left is $\log Z(x)$, a sum over every possible response, which you cannot compute. Here the shape of preference data saves us. Humans never gave absolute scores; they gave comparisons — "between A and B, A." The classical way to turn a comparison into a probability is the Bradley-Terry model.

$y_w$ is the response the human chose, $y_l$ the one they didn't, and $\sigma$ is the sigmoid. Put in words: push the two responses' score gap through the sigmoid and out comes the probability a human picks the winner — the bigger the gap, the closer that probability sits to one, exactly like a sports rating system.

The thing to notice about equation (4) is that it only takes the difference of rewards. Substitute equation (3) and $\beta\log Z(x)$ appears identically on both sides, so the subtraction cancels it outright. The term we couldn't compute vanishes because of how the problem was posed.