Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders

A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing.

Training language models to follow instructions with human feedback

Scaling up does not buy obedience

The InstructGPT paper opens with a flat denial: making language models bigger does not inherently make them better at following a user's intent (Abstract). Large models emit things that are untruthful, toxic, or simply unhelpful. The paper's diagnosis is not a capability gap but a mismatch of objectives.

What pretraining maximises is "predict the next token on a webpage from the internet". What we want is "follow the user's instructions helpfully and safely". The paper's word for the gap is misaligned (§1).

Aligned to what, then? The paper names three properties: helpful (it helps you solve your task), honest (it does not fabricate or mislead), harmless (it causes no physical, psychological or social harm) (§1, §3.6). Honesty, though, cannot be measured — you cannot read a model's beliefs from outside — so the paper measures truthfulness, whether the model's statements about the world are true, and says so plainly (§3.6).

Stage 1: collect demonstrations, fine-tune supervised (SFT)

The move is simple. Have labelers write demonstrations of the answer they want for a prompt, and fine-tune GPT-3 on them with supervised learning (§3.1, Step 1).

The scale is smaller than you would guess: about 13,000 training prompts for SFT (11,295 labeler-written plus 1,430 from API customers) (§3.2, Appendix A.3, Table 6). The mix of use cases is telling too — classification and QA are only about 18% of the distribution, while open-ended generation and brainstorming make up roughly 57% (Table 1, §4.1). That is simply not the shape public NLP datasets are good at.

One training detail contradicts the textbook. SFT runs for 16 epochs even though validation loss overfits after a single epoch — and yet more epochs kept improving both the reward-model score and human preference ratings (§3.5).

Stage 2: turn human rankings into a reward model

Demonstrations alone are not enough. Writing a good answer every time is expensive — and choosing which of two answers is better is far easier than writing one.

So labelers rank a set of model outputs for the same prompt. The paper shows them between $K=4$ and $K=9$ responses and turns each ranking into $\binom{K}{2}$ pairwise comparisons (§3.5). The RM training set is 33,000 prompts (Table 6).

The reward model itself is the SFT model with its final unembedding layer removed, taking a prompt and a response and returning one scalar (§3.5). The loss is:

Read left to right, it says: take a pair of responses a human judged, subtract the loser's score from the winner's, squash that difference into a probability, and penalise the model whenever that probability comes out low. Put in words, then, the loss does not care about the absolute size of a score at all. Only the gap between the two matters — add the same number to both and the loss does not move, so "80 against 60" and "20 against 0" are the same state as far as this equation is concerned.

Symbol by symbol: $x$ is the prompt, $y_w$ the response the human preferred, $y_l$ the one they did not, $r_\theta$ the scalar the reward model outputs, and $\sigma$ the sigmoid. All equation (1) says is score the winner above the loser. The sigmoid converts a score difference into "the probability a human picks that one" — a difference of zero gives 0.5, and larger differences approach 1.

Two implementation notes are spelled out. First, shuffling the $\binom{K}{2}$ comparisons into one pile and training on them one at a time overfits within a single epoch, because comparisons from the same prompt are strongly correlated. The fix is to treat all comparisons from a prompt as a single batch element, which is also cheaper — one forward pass per completion instead of one per pair (§3.5, footnote 3).