Instruction Tuning and RLHF from Scratch — How a Model Learns to Follow Orders

A pretrained model ignores your instructions not because it lacks the ability but because it was optimised for something else. Working only from the InstructGPT paper (Ouyang et al., 2022): the three stages — SFT, reward model, RL — down to the equations, the claim that a 1.3B model beat a 175B one in human evaluation, and the limits the authors themselves put in writing.

Training language models to follow instructions with human feedback

Scaling up does not buy obedience

The InstructGPT paper opens with a flat denial: making language models bigger does not inherently make them better at following a user's intent (Abstract). Large models emit things that are untruthful, toxic, or simply unhelpful. The paper's diagnosis is not a capability gap but a mismatch of objectives.

What pretraining maximises is "predict the next token on a webpage from the internet". What we want is "follow the user's instructions helpfully and safely". The paper's word for the gap is misaligned (§1).

Aligned to what, then? The paper names three properties: helpful (it helps you solve your task), honest (it does not fabricate or mislead), harmless (it causes no physical, psychological or social harm) (§1, §3.6). Honesty, though, cannot be measured — you cannot read a model's beliefs from outside — so the paper measures truthfulness, whether the model's statements about the world are true, and says so plainly (§3.6).

Stage 1: collect demonstrations, fine-tune supervised (SFT)

The move is simple. Have labelers write demonstrations of the answer they want for a prompt, and fine-tune GPT-3 on them with supervised learning (§3.1, Step 1).

The scale is smaller than you would guess: about 13,000 training prompts for SFT (11,295 labeler-written plus 1,430 from API customers) (§3.2, Appendix A.3, Table 6). The mix of use cases is telling too — classification and QA are only about 18% of the distribution, while open-ended generation and brainstorming make up roughly 57% (Table 1, §4.1). That is simply not the shape public NLP datasets are good at.

One training detail contradicts the textbook. SFT runs for 16 epochs even though validation loss overfits after a single epoch — and yet more epochs kept improving both the reward-model score and human preference ratings (§3.5).

Stage 2: turn human rankings into a reward model

Demonstrations alone are not enough. Writing a good answer every time is expensive — and choosing which of two answers is better is far easier than writing one.

So labelers rank a set of model outputs for the same prompt. The paper shows them between $K=4$ and $K=9$ responses and turns each ranking into $\binom{K}{2}$ pairwise comparisons (§3.5). The RM training set is 33,000 prompts (Table 6).

The reward model itself is the SFT model with its final unembedding layer removed, taking a prompt and a response and returning one scalar (§3.5). The loss is:

Read left to right, it says: take a pair of responses a human judged, subtract the loser's score from the winner's, squash that difference into a probability, and penalise the model whenever that probability comes out low. Put in words, then, the loss does not care about the absolute size of a score at all. Only the gap between the two matters — add the same number to both and the loss does not move, so "80 against 60" and "20 against 0" are the same state as far as this equation is concerned.

Symbol by symbol: $x$ is the prompt, $y_w$ the response the human preferred, $y_l$ the one they did not, $r_\theta$ the scalar the reward model outputs, and $\sigma$ the sigmoid. All equation (1) says is score the winner above the loser. The sigmoid converts a score difference into "the probability a human picks that one" — a difference of zero gives 0.5, and larger differences approach 1.

Two implementation notes are spelled out. First, shuffling the $\binom{K}{2}$ comparisons into one pile and training on them one at a time overfits within a single epoch, because comparisons from the same prompt are strongly correlated. The fix is to treat all comparisons from a prompt as a single batch element, which is also cheaper — one forward pass per completion instead of one per pair (§3.5, footnote 3).

Second, only 6B reward models are used. Training a 175B RM could be unstable and was less suitable as the value function for RL (§3.5). Since the loss is invariant to shifts in reward, the RM is finally normalised so labeler demonstrations average a score of zero.

Stage 3: update the policy with reinforcement learning

With a reward model in hand, the SFT model is fine-tuned with PPO using that scalar as reward (§3.1, Step 3). The environment is a bandit — one prompt appears, one response is returned, the reward model scores it, the episode ends (§3.5). The value function is initialised from the RM.

The objective being maximised is:

Put in words: a carrot, a stick, and a revision session. The large $\mathbb{E}$ only means "average this over many samples", $\phi$ is the bundle of weights being trained, and $\beta$ and $\gamma$ are dials for how hard the stick and the revision session pull. Chase the reward model's score (the carrot), pay a price for every step taken away from the SFT model you started from (the stick), and in the last term read some pretraining text again so the model does not forget what it could already do.

Three terms, where $\pi^{\mathrm{RL}}_\phi$ is the policy being learned and $\pi^{\mathrm{SFT}}$ is the SFT model it started from.

1. $r_\theta(x,y)$ — push up the reward model's score. This is the engine.
2. $-\beta\log(\pi^{\mathrm{RL}}_\phi/\pi^{\mathrm{SFT}})$ — a per-token KL penalty, charging the policy for drifting from the SFT model. Its stated purpose is to mitigate over-optimisation of the reward model (§3.5): the RM is only a stand-in for humans, and pursued to its limit it leads somewhere humans do not care about.
3. $\gamma\log\pi^{\mathrm{RL}}_\phi(x)$ — the log likelihood of pretraining data. With $\gamma=0$ you get plain "PPO"; positive gives "PPO-ptx", which is what the paper means by InstructGPT.

Why the third term exists leads straight into the next section.

What the paper showed

The headline is that a small model beat a large one under human evaluation. Outputs from the 1.3B InstructGPT were preferred to outputs from the 175B GPT-3 — over 100x fewer parameters, the same architecture, differing only in having been fine-tuned on human data (Abstract, §4.1). Size for size, 175B InstructGPT is preferred to 175B GPT-3 85±3% of the time, and to a few-shot-prompted 175B GPT-3 71±4% of the time (§4.1).

• Truthfulness: on TruthfulQA, roughly twice as many truthful and informative answers as GPT-3. On closed-domain tasks like summarisation, made-up information drops to 21% against 41% (§1, §4.2).
• Toxicity: about 25% fewer toxic outputs when prompted to be respectful. But bias on Winogender and CrowS-Pairs did not significantly improve (§1, §4.2).
• Regressions on public datasets: SQuAD, DROP, HellaSwag and WMT 2015 French–English all got worse than GPT-3. The paper names this the alignment tax, reports that mixing pretraining gradients in (PPO-ptx) removes most of it, and admits DROP, SQuADv2 and translation still lag GPT-3 (§4.2).
• Against FLAN and T0: InstructGPT is clearly preferred over 175B GPT-3 fine-tuned on those public task collections — a 73.4±2% winrate against the SFT baseline, versus 26.8±2% for T0 and 29.8±2% for FLAN (§4.1).
• Cost: 4.9 petaflops/s-days to train the 175B SFT model, 60 for PPO-ptx, against 3,640 for pretraining GPT-3. Alignment is orders of magnitude cheaper than pretraining (§5.1).

Limits and open questions

Sections 5.2 and 5.3 are unusually candid.

• This is not alignment to "human values". It is alignment to the preferences of about 40 contracted labelers, to the researchers who wrote the labeling instructions, and to what API customers find valuable. The paper states outright that it is not claiming researchers, the labelers hired, or API customers are the right source of preferences (§5.2).
• The labelers do not agree with each other. Inter-annotator agreement is 72.6±1.5% among training labelers and 77.3±1.3% for held-out ones (§3.4) — about a quarter of cases are contested. Worse, most comparisons were labelled by only one contractor, for cost reasons (§5.3).
• Neither fully aligned nor fully safe. Fabrication and toxic output remain. The greatest limitation named is that the models follow the user's instruction even when doing so could cause harm — prompted to be maximally biased, InstructGPT generates more toxic outputs than an equivalently-sized GPT-3 (§5.3).
• Simple failures persist: assuming a false premise is true, hedging at length on questions with a clear answer, degrading when an instruction carries several explicit constraints (§4.3). On the hedging, the paper's own guess is that instructing labelers to reward epistemic humility got picked up by the reward model.
• Almost entirely English — over 96% of the data (§3.3).

A note as of this writing in 2026 (mine, not the paper's): since this work, methods like DPO — which collapse the reward model and the RL stage into a single classification-style loss — have become widespread. Every number and claim above is from this paper alone; nothing from later work is mixed in.

How this shows up on the job

Of the three stages, the only one most teams actually run is the first. Building a reward model and an RL loop in-house belongs to organisations with a dedicated team and a standing labeling operation.

The knob names. In TRL, start at SFTTrainer with num_train_epochs, learning_rate, packing. Moving to preference learning: DPOTrainer's beta — the same role as $\beta$ here, deciding how far you may drift from where you started — and a frozen ref_model. Going all the way to RLHF: PPOTrainer's init_kl_coef and target. Whichever route you take, "how much KL from the reference model do I allow" is the knob that matters most.

Three traps. One: early stopping on validation loss. As §3.5 shows, instruction-following quality does not track it. Two: shuffling comparisons that came from the same prompt. Strongly correlated pairs become separate gradient steps and the RM overfits within one epoch (§3.5, footnote 3). Three: not measuring the alignment tax. Watch only instruction-following and your extractive QA and translation degrade quietly.

The design-review question is "why insert a reward model at all — why not use human judgement directly?" Two threads. First, RL needs a reward every episode; asking a human each time fails on both speed and cost, so the reward model is a fast approximation of human preference. Second, precisely because it is an approximation it can break, which is what the KL term in equation (2) is fencing in. A reward model is a proxy metric, and proxy metrics pursued hard enough drift off the real objective — say that and you have understood the design.

Making the supervised stage itself cheaper is the subject of the LoRA paper explained. And if "bad validation loss, good human ratings" is still uncomfortable, overfitting and evaluation is the piece to read first.

Summary

• Not following instructions is an objective problem, not a capability one — pretraining optimises the next token (§1)
• Three stages: SFT on demonstrations → a reward model from rankings (equation (1)) → PPO on the policy (equation (2)); the KL penalty fences in over-optimisation of the reward model
• The headline result: 1.3B beat 175B under human evaluation, and size-for-size the win rate was 85±3% (§4.1)
• The alignment tax is real. Mixing in pretraining gradients removes most of it, but DROP, SQuADv2 and translation never fully recover (§4.2)
• The alignment target is ~40 labelers, the researchers, and API customers — not "human values" (§5.2)

Original paper: Ouyang et al., "Training language models to follow instructions with human feedback" (arXiv:2203.02155, 2022)