Scaling Laws from Scratch — Why Making Models Bigger Makes Them Smarter (and When It Doesn't)

Scaling laws turn 'bigger is smarter' into an equation. This article walks from the Kaplan-era recipe to the Chinchilla paper (Hoffmann et al., 2022) — 400+ training runs, three independent estimation methods, and the 70B-vs-280B showdown — using only the paper itself as evidence.

Training Compute-Optimal Large Language Models

The budget is fixed. How do you spend it?

Suppose you have exactly 1,000 hours to prepare for an exam. You could master one massive textbook, or work through a thinner one plus a mountain of practice problems. Same total hours, very different outcomes depending on the split.

Training a large language model (LLM) has the same structure. Once you know how many GPUs you have and for how long, your total compute budget — measured in FLOPs, floating-point operations — is essentially fixed in advance. You can spend it on a bigger model or on more training data, but not both: a larger model burns more compute per token, so the same budget buys fewer tokens. And since frontier-scale training runs are effectively one-shot, the split has to be decided before you press go (§1).

The paper that tackled this allocation problem head-on is DeepMind's "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022), universally known as the Chinchilla paper. This article explains scaling laws from zero, using only that paper as its source.

The Kaplan era: "just make it bigger"

The story starts two years earlier. Kaplan et al. (2020) showed that a power law links a language model's parameter count to its loss: scale the model by orders of magnitude and the loss falls at a predictable rate. After that discovery, the field raced to build ever-larger models (§1, §2).

The Kaplan analysis went further and prescribed an allocation: given a 10× larger compute budget, grow the model 5.5× but the training tokens only 1.8× (§1). In other words, pour almost all of the extra budget into model size. And that is what happened: as the paper's Table 1 shows, GPT-3 (175B), Jurassic-1 (178B), Gopher (280B), and MT-NLG (530B) kept getting bigger while nearly all of them were trained on roughly the same ~300 billion tokens.

The Chinchilla paper's claim, in one sentence: that allocation was wrong, and today's large models are significantly undertrained (Abstract).

Turning the allocation problem into one equation

The paper first states the question as math. Treat the final pre-training loss $L$ as a function $L(N, D)$ of the parameter count $N$ and the number of training tokens $D$, then minimize it under a compute constraint $C$ (§1).

Underneath the notation this is a menu problem, which says: of every pairing of brain size and reading list you can afford, take the one that ends up predicting text best.

Reading the symbols one at a time: $N$ is the model's parameter count (the size of the brain), $D$ is the number of tokens it reads during training, and $C$ is the total compute available. The whole expression just says: among all $(N, D)$ pairs that cost exactly $C$ FLOPs, find the pair with the lowest loss. The functions $N_{opt}$ and $D_{opt}$ are the optimal model size and data amount for that budget.

A handy approximation makes the constraint concrete (§3.3):

$$C \approx 6ND$$

Total training compute is roughly "parameters × tokens × 6" (the 6 comes from 2 FLOPs per parameter for the forward pass plus twice that for the backward pass; Appendix F checks this carefully and finds the approximation accurate to within a few percent). Fix the budget $C$ and $N$ and $D$ become inversely proportional — which is exactly what makes this an allocation problem. Said in words: on a fixed budget, every parameter you add is paid for with tokens the model will never get to read.