Scaling Laws from Scratch — Why Making Models Bigger Makes Them Smarter (and When It Doesn't)

Scaling laws turn 'bigger is smarter' into an equation. This article walks from the Kaplan-era recipe to the Chinchilla paper (Hoffmann et al., 2022) — 400+ training runs, three independent estimation methods, and the 70B-vs-280B showdown — using only the paper itself as evidence.

Training Compute-Optimal Large Language Models

The budget is fixed. How do you spend it?

Suppose you have exactly 1,000 hours to prepare for an exam. You could master one massive textbook, or work through a thinner one plus a mountain of practice problems. Same total hours, very different outcomes depending on the split.

Training a large language model (LLM) has the same structure. Once you know how many GPUs you have and for how long, your total compute budget — measured in FLOPs, floating-point operations — is essentially fixed in advance. You can spend it on a bigger model or on more training data, but not both: a larger model burns more compute per token, so the same budget buys fewer tokens. And since frontier-scale training runs are effectively one-shot, the split has to be decided before you press go (§1).

The paper that tackled this allocation problem head-on is DeepMind's "Training Compute-Optimal Large Language Models" (Hoffmann et al., 2022), universally known as the Chinchilla paper. This article explains scaling laws from zero, using only that paper as its source.

The Kaplan era: "just make it bigger"

The story starts two years earlier. Kaplan et al. (2020) showed that a power law links a language model's parameter count to its loss: scale the model by orders of magnitude and the loss falls at a predictable rate. After that discovery, the field raced to build ever-larger models (§1, §2).

The Kaplan analysis went further and prescribed an allocation: given a 10× larger compute budget, grow the model 5.5× but the training tokens only 1.8× (§1). In other words, pour almost all of the extra budget into model size. And that is what happened: as the paper's Table 1 shows, GPT-3 (175B), Jurassic-1 (178B), Gopher (280B), and MT-NLG (530B) kept getting bigger while nearly all of them were trained on roughly the same ~300 billion tokens.

The Chinchilla paper's claim, in one sentence: that allocation was wrong, and today's large models are significantly undertrained (Abstract).

Turning the allocation problem into one equation

The paper first states the question as math. Treat the final pre-training loss $L$ as a function $L(N, D)$ of the parameter count $N$ and the number of training tokens $D$, then minimize it under a compute constraint $C$ (§1).

Underneath the notation this is a menu problem, which says: of every pairing of brain size and reading list you can afford, take the one that ends up predicting text best.

Reading the symbols one at a time: $N$ is the model's parameter count (the size of the brain), $D$ is the number of tokens it reads during training, and $C$ is the total compute available. The whole expression just says: among all $(N, D)$ pairs that cost exactly $C$ FLOPs, find the pair with the lowest loss. The functions $N_{opt}$ and $D_{opt}$ are the optimal model size and data amount for that budget.

A handy approximation makes the constraint concrete (§3.3):

$$C \approx 6ND$$

Total training compute is roughly "parameters × tokens × 6" (the 6 comes from 2 FLOPs per parameter for the forward pass plus twice that for the backward pass; Appendix F checks this carefully and finds the approximation accurate to within a few percent). Fix the budget $C$ and $N$ and $D$ become inversely proportional — which is exactly what makes this an allocation problem. Said in words: on a fixed budget, every parameter you add is paid for with tokens the model will never get to read.

Measuring the terrain with 400 models — three estimation methods

The catch is that nobody knows the shape of $L(N, D)$. So the paper trained over 400 models, from 70 million to over 16 billion parameters, on 5 billion to hundreds of billions of tokens, and estimated the optimal allocation three independent ways (Abstract, §3).

Method 1: fix model sizes, vary training length (§3.1). Each model size is trained for four different horizons. Taking the envelope of "lowest loss achieved at each FLOP count" and fitting power laws $N_{opt} \propto C^a$, $D_{opt} \propto C^b$ gives $a = 0.50$, $b = 0.50$.

Method 2: IsoFLOP profiles (§3.2). Now fix the compute budget — nine values from $6 \times 10^{18}$ to $3 \times 10^{21}$ FLOPs — and vary only the model size within each budget. Plotting final loss against size produces a U-shaped valley per budget; the valley floor is the optimal size for that budget. Fitting parabolas to the valleys and power laws across budgets gives $a = 0.49$, $b = 0.51$.

Method 3: model the loss function itself (§3.3). Fit all final losses with this functional form:

The shape is a floor plus two penalties, which says in words: how well a model can possibly do is capped by how unpredictable language is to begin with, then worsened by having too small a brain and by not having read enough.

Each term has a meaning. $E$ is the loss that would remain even for an ideal generative process — the entropy of natural text, its inherent unpredictability. The second term is the penalty for having only finitely many parameters $N$; the third is the penalty for training on finite data $D$ without reaching convergence. The fit yields $E = 1.69$, $A = 406.4$, $B = 410.7$, with exponents $N^{0.34}$ and $D^{0.28}$ (§D.2). The allocation implied by this fit is $a = 0.46$, $b = 0.54$.

The conclusion: grow model and data in equal proportion

Three methods, different models, different fitting procedures — nearly the same answer. When the compute budget grows, model size and training data should grow in equal proportion ($a \approx b \approx 0.5$). Double the model, double the data (Abstract, §3.4). That flatly contradicts Kaplan's $a = 0.73, b = 0.27$ (Table 2).

Why the disagreement? The paper points to two main causes (§2). First, Kaplan et al. used a fixed token count and a fixed learning-rate schedule for every model. The Chinchilla paper shows that the cosine learning-rate decay must be matched to the actual training horizon: stretch the cosine cycle more than 25% past the real number of steps and final performance clearly degrades (Appendix B). Judging short runs by their mid-schedule loss systematically understates them. Second, scale: Kaplan's runs were mostly models under 100M parameters, while the majority of runs here exceed 500M (§2).

In plain words: if your budget quadruples, double the model and double the data. The same exponents reappeared when the experiments were rerun on C4 and on GitHub code (Appendix C). The paper even staged a head-to-head at $10^{21}$ FLOPs: a 4.74B-parameter model sized per Kaplan's rule versus a 2.80B model sized per their own — and the smaller model won (Appendix D.4).

The proof: 70B Chinchilla beats 280B Gopher

If the estimates are right, then the budget spent on Gopher (280B parameters, 300B tokens) — $5.76 \times 10^{23}$ FLOPs — should have bought a model of 40–70B parameters (§4). The authors picked the top of that range and trained a 70B-parameter model on 1.4 trillion tokens: one quarter of Gopher's size, four times its data, the same compute. That model is Chinchilla. Its architecture is essentially Gopher's; the differences are minor — AdamW instead of Adam, a slightly modified tokenizer, and a float32 copy of the weights kept in the optimizer state (§4.1).

The results were one-sided. On MMLU, a 57-subject exam-style benchmark, Chinchilla averaged 67.6% against Gopher's 60.0% — a 7.6-point jump that even leapfrogged the 63.4% that expert forecasters had predicted as the June 2023 state of the art (§4.2.2). On the 62-task BIG-bench average: 65.1% versus 54.4% (§4.2.4). On RACE-h reading comprehension, an improvement of more than 10 points (§4.2.3). Chinchilla also beat Gopher on all 19 subsets of The Pile, though the paper itself cautions that with 4× more training data, train/test overlap may flatter those language-modeling numbers (§4.2.1).

Don't miss the inference angle. Same budget, better scores, and a model one quarter the size: everything downstream — inference, fine-tuning, memory footprint — gets substantially cheaper (§4). A one-time allocation decision at training time pays dividends across the model's entire serving life.

What the paper does not establish

The authors are explicit about the limits (§5). The large-scale comparison rests on a single pair — Chinchilla versus Gopher — with no tests at intermediate scales. The whole analysis assumes power laws, yet the frontier shows slight concavity at high budgets, which means the optimal model size may still be overestimated (Appendix E). Every run used less than one epoch of data, so the repeated-data regime is untested. The claim that further data scaling only helps if the data is high-quality is flagged by the authors themselves as speculative. On bias and toxicity, Chinchilla came out roughly at Gopher's level, as expected for a model trained on the same dataset (§4.2.7).

How this is used on the job

Scaling laws earn their keep when you are planning a pre-training or continued-pre-training run — they are the calculator an ML engineer reaches for before writing the GPU procurement request.

• Work backwards from the budget: GPU-hours × chip FLOP/s × utilization gives you $C$; plug it into $C \approx 6ND$ and the feasible $(N, D)$ pairs fall out. Dividing the numbers in the paper's Table 3 puts the optimum near ~20 tokens per parameter (a 10B model wants ~205B tokens; 70B wants 1.4T). That ratio is a practical rule of thumb derived from the paper's own tables.
• The scheduler trap: make the cosine decay horizon (the max_steps / T_max-style setting in your training framework) match the steps you will actually run. In the paper, overshooting the cycle length by more than 25% measurably hurt final performance (Appendix B). "Set it long and stop early" is a self-inflicted wound.
• Inference cost is outside the equation: the paper optimizes training FLOPs only. If you will serve the model at scale, the inference bill needs its own analysis — that is a practical judgment beyond the paper's scope.
• If it comes up in an interview: "Your compute budget doubles — what do you scale?" The Chinchilla answer: model and data each by $\sqrt{2}$ (about 1.4×). Bonus points for explaining how the learning-rate-schedule issue produced the older, model-heavy Kaplan prescription.

Takeaways

• A scaling law is an empirical power-law relationship between loss, model size, and data; the allocation problem is a constrained minimization under $C \approx 6ND$
• With 400+ runs and three independent methods, the Chinchilla paper estimates that model and data should scale in equal proportion ($a \approx b \approx 0.5$) — overturning the model-heavy Kaplan recipe
• As proof, Chinchilla (70B × 1.4T tokens) beat the 4×-larger, same-budget Gopher on nearly every evaluation
• Caveats from the paper itself: one large-scale comparison, sub-one-epoch training only, and a possible remaining overestimate of optimal model size

For the Transformer machinery underneath all of this, see the Attention Is All You Need walkthrough; for the "small but strong" lineage that followed, see the LLaMA family guide. And for the infrastructure that actually pushes trillions of tokens through a cluster, see how distributed training works.