Scaling Skepticism — A Genealogy of the "Just Make It Bigger" Critique

A fair accounting of the case against "more parameters, more data" — data exhaustion, the reasoning wall, and the world-model dispute. The evidence for scaling gets the same scrutiny as the evidence against it, and every claim comes with the observation that would settle it.

Training Compute-Optimal Large Language Models

The sound barrier was real. It was also crossed

In the propeller era, making an aircraft faster was a solved problem: build a bigger engine. Records climbed for years on exactly that recipe — and then, as speeds approached Mach 1, they stopped climbing. Engineers split three ways. There is no barrier, we just need more power. There is a barrier, and humans will never fly faster. There is a barrier, but it comes from propellers and straight wings, and a different design would clear it.

The third camp was right. The barrier was real and it was crossed — not by a bigger engine, but by swept wings and jet propulsion.

The argument now running through large language models has the same shape. "More parameters and more data make it smarter" held up remarkably well for several years after 2020. And now, from several directions at once, people are arguing that the recipe won't get us where we're going. This article lays those arguments out fairly. Not to mock the skeptics or bury the believers, but to sort out which claim rests on which evidence — and what observation would settle each one.

First, state the claim being attacked — precisely

Before reading the skeptics, let's pin down the target. Vague targets are easy to hit.

A scaling law is not the woolly assertion that bigger is smarter. It is the quantitative claim that loss falls smoothly as a power of parameter count and data size. Hoffmann et al.'s Chinchilla paper fit it to this form:

$L$ is the loss (how hard the next word is to guess), $N$ the parameter count, $D$ the number of training tokens. $E$ is the term that never goes away no matter how big you build, standing in for the irreducible unpredictability of language itself. $A/N^{\alpha}$ is the penalty for the model being too small; $B/D^{\beta}$ is the penalty for having too little data. So the equation decomposes underperformance into "not enough model," "not enough data," and "impossible in principle." In the paper's fit, $\alpha$ and $\beta$ both came out near 0.3 — close enough to each other that when the budget grows, the optimum is to grow model and data at the same rate. That is what the "Chinchilla rule" actually is; the derivation is in Scaling Laws From Scratch.

The strongest evidence on the pro-scaling side is that the equation extrapolated. The GPT-4 technical report describes fitting the law to small models trained with up to 10,000× less compute, then predicting the production model's final loss ahead of time. That is a serious engineering result — it is the same kind of thing as computing a bridge's strength before you build it. The intellectual backbone is Rich Sutton's "The Bitter Lesson" (2019): across AI's history, methods that hand-encode human knowledge have lost, in the long run, to methods that let compute do the work.

Here is where the debate most often goes off the rails. Three separate claims travel under one banner: (1) more input lowers loss, (2) lower loss raises capability, (3) rising capability eventually reaches human-level intelligence. Scaling laws demonstrated (1). Claim (2) holds roughly, empirically, most of the time. Claim (3) isn't a finding at all — it's a bet. Skeptics usually aim at (2) or (3); defenders usually answer with the strength of (1). A lot of the apparent disagreement is just that mismatch, and keeping the three claims apart is the most useful habit you can bring to any argument in this space.

And the equation never promised unbounded improvement. A power law that looks like a straight line on a log-log plot means, on a linear axis, that every equal increment of improvement costs a constant multiple more input. Believers and skeptics are staring at the same graph.

Let's put a number on that multiple, using Chinchilla's exponent.

alpha = 0.34          # the N-side exponent Chinchilla fit
# At the optimal allocation, N grows as C^0.5 (D grows at the same rate)
def compute_multiplier(shrink):        # shrink=0.5 -> halve the model-side loss term
    n_mult = shrink ** (-1 / alpha)    # required parameter-count multiple
    return n_mult ** 2                 # since N ∝ C^0.5, compute is that squared

print(round(compute_multiplier(0.5)))  # ≈ 59

Halving just the model-side loss term costs roughly 59× the compute. Halving it again costs another 59×. This isn't a criticism; it's what the law says when you read it literally. The difference between a believer and a skeptic is whether that staircase looks climbable or looks like a wall. Below are the three obstacles people have placed on the steps — each a different kind of claim, and each falsified by a different observation.