What Happens When You Make It Bigger — Scaling and Pre-training

Textbook › Part III Deep Learning: what assumption you add

CHAPTER 8

What Happens When You Make It Bigger — Scaling and Pre-training

★ MEMBER12 min

Everything through Chapter 7 built the tools for making one model and training it. This chapter's question sits a level up: when more compute becomes available, what do you spend it on?

There are two choices: a bigger model (parameters $N$ ) or more data (tokens $D$ ). Both are available; both at once is not, because the budget is fixed.

So first, derive the budget equation by counting it ourselves. Then solve the allocation.

1. Counting the compute

The weights in one Transformer block were counted in Chapter 7 (with $d_{ff} = 4d$ ):

	weights
attention ( $W_Q,W_K,W_V,W_O$ )	$4d^2$
FFN (two layers)	$8d^2$
one block	$12d^2$

From there, the cost per token:

Forward: one multiply and one add per weight — 2 FLOPs — so $2N$ in total.
Backward: gradients flow to both the inputs and the weights, roughly twice the forward pass, so $4N$ (Chapter 3, equations (3) and (4)).

Together, $6N$ per token, and over $D$ tokens:

C \;\approx\; 6ND

(1)

This equation uses nothing beyond this book's arithmetic. Putting it to work:

$N$	$D$	$C = 6ND$
$10^{8}$	$10^{9}$	$6.0\times10^{17}$
$10^{9}$	$2\times10^{10}$	$1.2\times10^{20}$
$7\times10^{10}$	$1.4\times10^{12}$	$5.9\times10^{23}$

2. Where $6ND$ breaks

Equation (1) counts only the computation that passes through weights. The $T \times T$ attention table from Chapter 6 is not a weight, so it is not in there. A division says when it starts to matter.

Per layer per token, the weight term is $2 \times 12d^2 = 24d^2$ , and the table term ( $QK^{\top}$ and $PV$ ) is $2 \times (2Td) = 4Td$ :

\frac{\text{table term}}{\text{weight term}} \;=\; \frac{4Td}{24d^2} \;=\; \frac{T}{6d}

(2)

Once the context length $T$ passes $6d$ , the table outweighs the weights.

$d$	$6d$	$T=1$ k	$T=4$ k	$T=16$ k	$T=64$ k
512	3,072	0.33	1.33	5.33	21.33
1,024	6,144	0.17	0.67	2.67	10.67
4,096	24,576	0.04	0.17	0.67	2.67

At $d = 512$ and $T = 4$ k the table already costs 1.33× the weights. $C \approx 6ND$ is a short-context approximation; in long-context models Chapter 6's $T^2$ comes back.

3. Splitting the budget

Fix $C$ and choosing $N$ fixes $D = C/(6N)$ . All that remains is to pick the $N$ that minimises the loss — which requires knowing how the loss depends on $N$ and $D$ .

Here we assume a form:

L(N, D) \;=\; \frac{A}{N^{a}} \;+\; \frac{B}{D^{b}} \;+\; E

(3)

The first term is the cost of the model being small, the second of the data being scarce, and $E$ is a floor neither can remove (the role $H(p)$ played in Chapter 0-4).

$A, B, E, a, b$ are not measured in this book. What follows uses $A=B=400$, $E=1.0$ and $a=b=0.34$ as placeholders, to see the shape — not values that hold for real models. Measured values exist in the literature and are not quoted here; only the claim in section 5 is cited, with its source.

FIG 1The horizontal axis is N (right = bigger model, less data), the vertical the assumed loss. Move the budget C and the optimum (gold) moves. The exponents a and b are knobs too: only when a equals b does the optimum land at D/N = 1. You can see exactly which assumption the "scale them equally" conclusion rests on

Solving at $C = 10^{21}$ under that assumption gives $N = 1.29\times10^{10}$ , $D = 1.29\times10^{10}$ and $D/N = 1.00$ . Varying the budget moves the optimum like this.

$C$	optimal $N$	optimal $D$
$10^{19}$	$1.29\times10^{9}$	$1.29\times10^{9}$
$10^{21}$	$1.29\times10^{10}$	$1.29\times10^{10}$
$10^{23}$	$1.29\times10^{11}$	$1.29\times10^{11}$

Ten times the budget, $\sqrt{10} = 3.16$ times each. They grow equally.