Test-Time Scaling — How Models Get Better by Thinking Longer

The same model scores higher when you let it think longer. This article builds the idea from scratch: chain-of-thought as purchased compute steps, self-consistency by majority vote, verifiers that pick the winner, and o1-style models that learned the thinking itself — and what it means for compute to shift from training to inference.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

The long think and the snap answer

People spend more time on harder problems. A chess player will sit motionless for ten minutes over a difficult position, then answer "what's 3 plus 4?" without pausing. Matching effort to difficulty is so ordinary that we never notice we're doing it.

Language models spent years without that ability. Whether you ask "what's 1 plus 1?" or "solve this open problem," producing one token costs exactly the same amount of arithmetic, because the layer count is fixed. A 24-layer model runs 24 layers' worth of computation on any input and emits the next word. It cannot think harder, and it cannot coast.

Picture an exam instead. If the instructions say "no working — write the answer only," you can solve whatever you can do in your head, and nothing more. Hand you scratch paper and suddenly three-digit multiplication is within reach. Your ability didn't change; you were given somewhere to work. Test-time scaling is the practice of handing that scratch paper to a language model — and the paper turns out to be the model's own stream of output tokens.

Compute has two places to live

The cost of a large model splits by when you pay it. Training compute is estimated from parameter count $N$ and training tokens $D$ roughly as:

Read it as: total training cost is a fixed number of operations per parameter, per token. The constant 6 comes from counting the multiply-adds in the forward and backward passes together. Spelled out in words: $N$ is how many knobs the model has and $D$ is how much text it was shown, which says that doubling either one doubles the electricity bill for training. Budgeting against this equation is exactly what scaling laws are about.

Inference compute, by contrast, is governed by the number of tokens $T$ you push through:

Read it as: about two operations per parameter per token — smaller than training's constant because there is no backward pass. Put in words, $T$ counts every token read and written, so the bill grows with how much the model writes even though the weights never change.

The decisive difference is how many times you pay. Training is paid once and amortized across every user forever; inference is paid on every single request. That asymmetry is why the industry spent years trying to shrink inference — quantization, distillation, KV caching. Test-time scaling takes the opposite bet: deliberately inflate $T$, and buy accuracy with the extra compute.

Why emitting tokens makes a model smarter

This is the crux. When a Transformer produces one token, the compute available to it is capped at "layers × cost per layer." One token is one step of computation, and no more. A problem that genuinely requires several stages of reasoning does not fit inside that one step.

But if the model writes an intermediate result into its output, it can read that result back as input when producing the next token. The moment it writes, two things happen at once: the intermediate value is parked in external memory, and one more computation step is unlocked. Write ten tokens and you have effectively bought ten extra steps.

That is all chain-of-thought (CoT) really is. "Let's think step by step" is not a motivational phrase — it's a purchase order for additional compute steps. Which parts of the CoT paper (Wei et al., 2022) were actually demonstrated, and which parts of prompt folklore were never tested, is the subject of the science of prompt engineering. The finding that matters here is that CoT is emergent above a certain model scale. Ask a small model to show its working and it writes plausible-looking broken working, and accuracy drops. Scratch paper is useless if you have nothing to write on it.

A single chain snaps in the middle

CoT's weakness is that there is only one reasoning path. Generation is stochastic: at each step the model draws a token from a distribution. Get a sign wrong at step three and the answer is wrong no matter how careful the remaining steps are — and it will be wrong in a confident tone of voice.

Take a concrete case: "You save 30 dollars a month and spent 120 dollars along the way. What's the balance after 18 months?" There are several valid routes — total the 18 months first and then subtract, or track the running balance month by month. Every route reaches the same answer, but which route the model takes is decided at sampling time, by chance. And if it fumbles "30 × 18" as 520 somewhere along that route, everything after it is built on the error. A person would sanity-check the multiplication. CoT accepts whatever it just wrote and moves on.

How much variation there is comes down to temperature. Near zero, the model always takes the highest-probability token, so running it a hundred times returns the same answer a hundred times. Raise it and the distribution flattens, letting second- and third-choice paths surface. Use the figure below to feel how the distribution sharpens and flattens as you drag the temperature — that flatness is the fuel every technique in this article runs on.

Sample sixteen times at temperature 0 and you get sixteen copies of one answer: sixteen times the bill for one answer's worth of information. Without diversity, solving repeatedly buys you nothing. Push temperature too high and the working itself falls apart. That tension is the first knob.

Solve it many times and take a vote

Once you can sample multiple paths, the next move is obvious: generate N solutions and take a majority vote on the final answer. This is Self-Consistency (Wang et al., 2022). The justification is marginalization — what you care about is the probability of the answer $a$, not the path $r$ that produced it.

Read it as: the probability of answer $a$ to problem $x$ is the sum of the probabilities of every route $r$ that arrives at $a$ — which says that an answer's standing comes not from the single likeliest route to it, but from how many routes land on it once you add them all up. Greedy CoT decoding looks at only the single highest-probability route in that sum and reports its answer. Different routes give different answers, and it counted exactly one vote.