Making It Follow Instructions — Instruction Tuning and Preference Optimization

Textbook › Part III Deep Learning: what assumption you add

CHAPTER 9

Making It Follow Instructions — Instruction Tuning and Preference Optimization

★ MEMBER21 min

Chapter 8 took us as far as splitting a budget between $N$ and $D$ . What comes out of that is a machine that continues text. Feed it a question and you get the most plausible continuation.

But "plausible continuation" is not "useful answer." This chapter looks at the two stages that follow pre-training — instruction tuning and preference optimization — from the loss function's side.

Neither one is a new mechanism. Both only change where the targets are placed. Let's confirm that with equations.

1. What pre-training never made a target

Section 6 of Chapter 8 counted targets: 999 from a 1,000-token document. Every one of them asks "what is the next token?"

Not one of them asks "which answer is better?"

This is not about capability. As said since Chapter 2, what is not in the loss is not learned. The pre-training loss was this.

L_{\text{pre}} \;=\; -\sum_{t} \log p(x_t \mid x_{<t})

(1)

"Was this useful to the reader?" does not appear. So when the continuation of a question turns out to be

another question in the same format (which is what a problem set contains)
"…which appeared on last year's exam"
a list of multiple-choice options

nothing has gone wrong as far as the loss is concerned. The training data really did continue that way, so as a job of fitting a distribution, it is correct.

A pre-trained model fails to follow instructions not because it cannot, but because following was never once a target. What is missing is not capability; it is targets.

2. Stage one: instruction tuning — same loss, different place to count

Instruction tuning (SFT, supervised fine-tuning) does exactly two things.

Make the data pairs of ⟨instruction, response⟩
Take the loss on the response only (do not count the instruction)

The loss itself stays the cross entropy of Chapter 4 of Part 0 — no new equation appears. What changes is where you count.

Where the gradient goes if you don't

Let the instruction be $P$ tokens and the response $R$ tokens. Taking the loss over every token leaves the response with a share of $R/(P+R)$ .

instruction $P$	response $R$	share of the response	with masking
200	50	20.0%	100%
500	20	3.8%	100%
1,000	200	16.7%	100%
100	100	50.0%	100%

For a pair with $P=500$ , $R=20$ — a common shape — counting everything sends 96.2% of the gradient into continuing the instruction text. That is pre-training again, not practice at following instructions.

Same data, same loss equation: change where you count and the content of the learning changes. That is why the instruction side is masked out.

How many targets are there

Look at the orders of magnitude. With 50,000 pairs and responses of 200 tokens on average, the target count is this.

5\times10^{4} \times 200 \;=\; 1\times10^{7}

(2)

Chapter 8's example pushed $D = 10^{11}$ tokens (≈ the same number of targets) through pre-training. The ratio is $10^{-4}$ .

SFT has one ten-thousandth of pre-training's targets. This is not the stage where knowledge is added. It is the stage where the model is made to choose which part of the distribution it already has to emit. Do not expect the same thing from two stages four orders of magnitude apart.

3. What SFT leaves out — it never teaches an order

Cross entropy is an instruction that says "match the distribution of the responses you were shown." That is where the weakness is.

Suppose one good response and one bad response are shown for the same instruction. The frequency in the data is 1:1. Maximum likelihood obeys faithfully and settles at 0.5 / 0.5. Draw one sample and the chance of the good one is 0.5 — the same as guessing.

draws	chance the bad one appears at least once
1	50.0%
3	87.5%
5	96.9%

That is $0.5^{n}$ turned around. SFT is imitation, so the average of what you showed comes back out. Whatever got mixed into the data comes out mixed.

There are two ways to fix it.

Delete the bad responses from the data. But you cannot delete what you cannot articulate. Stiff phrasing, conclusion buried, answering what was not asked — every writer draws those lines differently.
Ask only which one is better. No response has to be written out; two are put side by side and one is picked. What you get is not the text of the good response but an order.

The second is the doorway to preference optimization. The problem is how to turn an order into a loss.

4. Turning comparisons into scores — Bradley–Terry

Comparisons are all we have, so start by deciding how comparisons would come out if scores existed. Put the probability that response $A$ is preferred over $B$ as follows.

\begin{aligned} P(A \succ B) &\;=\; \sigma\!\left(r_A - r_B\right) \\ \sigma(z) &\;=\; \frac{1}{1+e^{-z}} \end{aligned}

(3)

The assumption is that the difference of scores is the logit (the Bradley–Terry model). With it in place, $r$ can be fitted from comparison data by logistic regression.

The correspondence between difference and win rate is computable.

reward gap $\Delta r$	win rate
0	50.0%
0.5	62.2%
1	73.1%
2	88.1%
3	95.3%
5	99.3%

It reads backwards too. A gap that makes people agree 70% of the time is $\ln(0.7/0.3) = 0.847$ ; 90% agreement is 2.197. Raising agreement moves the reward gap only slowly.

What is and is not determined about $r$

Under this assumption the origin of $r$ is not determined: adding the same constant everywhere leaves every difference unchanged. Measured: $r_A=1.2$ , $r_B=0.3$ gives 0.710950, and adding $+7.5$ to both still gives 0.710950.

The scale, on the other hand, matters. Doubling the gap gives 0.858149.

A number out of a reward model is not an absolute quality. Only the order and the size of the gaps carry meaning. "This model's reward was 3.2" says nothing without naming what it is being compared against.

5. Maximizing reward alone breaks it — the second knob

Now that scores exist, maximizing them looks like the whole job. Try it and the trouble shows.

The policy that maximizes $E_\pi[r]$ alone is the one that emits the top-reward response every time. As a distribution it collapses to a point. And $r$ was estimated from human comparisons, so it drifts to wherever the estimate is loosest. That is not searching for good responses; it is searching for holes in the reward model.

So add a second term, penalizing departure from a reference policy $\pi_{\text{ref}}$ (the model that finished SFT).