Chain-of-Thought from Scratch — What Changes When You Make a Model Show Its Work

Why does adding "think step by step" make a model solve problems it otherwise fails? A ground-up tour of few-shot CoT, zero-shot CoT and self-consistency — the serialization argument for why it works, the conditions where it doesn't, and how it relates to today's reasoning models.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Mental math versus paper

Ask someone for 23 × 47 out loud and most people stall. Hand them a pen and paper and almost everyone gets it. Nobody got smarter in between — they just gained somewhere to put the intermediate numbers. Write, re-read, move on: that loop carries a calculation further than working memory alone can hold it.

Much the same thing happens to large language models. A problem they get wrong when told "just give the answer" becomes solvable when you add "show your working." The phenomenon, and the prompting pattern that elicits it, is chain-of-thought (CoT). The 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models gave it the name.

What makes this strange is that not a single byte of the model's weights changed. You reworded the request, and the set of solvable problems grew. This article walks down from the analogy to the mechanism, the code, and the ways it goes wrong.

The actual artifact: few-shot CoT

What the original paper did is almost embarrassingly simple. It changed the worked examples in the prompt, and nothing else.

[standard]
Q: There are 3 cars in the lot. 2 more arrive. How many?
A: 5

[chain-of-thought]
Q: There are 3 cars in the lot. 2 more arrive. How many?
A: There were 3 to start. 2 more arrived, so 3 + 2 = 5. The answer is 5.

Put two to eight examples in that style ahead of the real question and the model imitates the format, writes out its steps, and — as a consequence of writing them — gets more answers right. The examples don't need to be related to the real question. All that has to transfer is the shape of an acceptable answer.

There are even reports that the effect partly survives when the arithmetic inside the exemplars is wrong. The model is imitating the behaviour of proceeding in steps more than it is imitating correct answers.

Skipping the examples: zero-shot CoT

Writing exemplars is tedious. The following year, Large Language Models are Zero-Shot Reasoners showed you can drop them entirely and append one line to the question:

Q: (the problem)
A: Let's think step by step.

That single sentence pushes the output distribution out of the "state a conclusion immediately" region and into the "lay out a procedure" region.

The paper adds a practical trick that still earns its keep: do it in two passes. The first call produces only the reasoning. Feed that whole output back in, append "Therefore, the answer is", and let the second call emit the answer alone. Parsing becomes far more reliable.

Why it works: trading depth for length

Two mechanisms, one deeper than the other.

First, whatever the model writes comes back as input. Every time an LLM emits a token it re-reads everything before it — your prompt plus everything it has already written. Writing out intermediate steps is literally scribbling on a scratchpad the model can read. Once "3 + 2 = 5" exists as text, every later token can pull that 5 straight out of context through attention. Unwritten, that 5 lingers vaguely in some activation, or never gets computed at all.

Second, and more fundamentally, the compute per token is fixed. A Transformer is $L$ stacked layers, and no matter what you feed it, the number of serially dependent computation steps available to produce one token tops out at roughly $L$. But plenty of problems are inherently serial: you cannot start step k until step k−1 has landed. Multiplication with carries, sequential updates to a list, multi-hop deduction. If a problem needs more serial stages than $L$, one forward pass cannot reach the answer — the depth simply isn't there.

CoT routes around exactly this.

The left side is the computational depth available when you demand the answer in one shot; the right side is the depth available when the model writes $T$ intermediate tokens first. $L$ is the layer count, fixed the day the model was built. $T$ is the generation length, which you control. You are buying more computation without a bigger model, by making the output longer.

The intuition has theoretical backing. Work such as Chain of Thought Empowers Transformers to Solve Inherently Serial Problems shows that, under standard assumptions, a constant-depth Transformer given enough chain-of-thought steps can handle classes of problems it cannot handle in a single forward pass. CoT is better understood as changing how compute is spent than as teaching the model how to think.

Seen this way, CoT was the entrance to test-time scaling: instead of spending more on training, spend more at inference. It was the first and cheapest version of that trade.

Self-consistency: solve it many times and vote

CoT is brittle. One slip halfway through and the model rides the mistake to a confident wrong answer — serial procedures fail totally at a single bad step, same as human long division.

Self-consistency is the remedy, and it is almost insultingly simple: solve the same problem several times and take a majority vote over the final answers only.

$N$ is how many times you solve it, $a_i$ is the final answer from run $i$, and $\mathbb{1}[\cdot]$ is 1 when the condition holds and 0 otherwise. In words: line up the $N$ answers and keep whichever one shows up most.

It works because there are many roads to the right answer, but wrong answers scatter. Different samples word things differently and take different valid routes, yet correct ones land on the same value; errors land wherever the particular slip took them. Count on the answer side and the correct value piles up.

Which makes sampling temperature load-bearing. At temperature 0 (greedy decoding), calling the model $N$ times returns the identical text $N$ times, so you must turn the temperature up to diversify the routes. Push it too far and the answers scatter along with the routes, and no pile forms.

The implementation is this much:

from collections import Counter

def self_consistency(model, question, n=8, temperature=0.7):
    prompt = f"{question}\nThink step by step, then end with 'Answer: X'."
    answers = []
    for _ in range(n):
        text = model.generate(prompt, temperature=temperature)  # a different route each time
        answers.append(extract_final_answer(text))              # keep only what follows 'Answer:'
    return Counter(answers).most_common(1)[0][0]                # the mode wins

The cost is exactly $N$×: you are buying accuracy with tokens. It also needs answers that normalize cleanly, since free-form prose defeats the vote through surface variation — so it suits numeric or multiple-choice outputs.

Know the failure conditions first

Small models. A central observation of the original paper was that the benefit depends on model scale. Ask a small model to show its work and you often get text with the right shape and broken content, scoring worse than answering directly. Format is imitable long before reasoning is.

Tasks that aren't math or symbol manipulation. To CoT or not to CoT?, which aggregates a large number of experiments, reports that the gains concentrate on math and symbolic tasks and are small for commonsense or knowledge recall. A device that buys serial computation has nothing to sell to a task that needs none.

Treating the written reasoning as an explanation. Language Models Don't Always Say What They Think showed that when an answer is steered by an irrelevant cue — the ordering of the answer options, say — the chain-of-thought never mentions that cue and instead supplies a plausible-sounding rationale for the biased answer. CoT is generated text, not a transcript of cognition.

Over-constrained output formats. A JSON schema permitting only {"answer": ...} leaves nowhere to reason, which is effectively CoT-off. If you want the reasoning, add a reasoning field and put it before answer. JSON keys are generated in order, and generation order is computation order.

How this relates to reasoning models

The "reasoning models" that arrived from 2024 onward are the next chapter of this story. Instead of asking for chain-of-thought in the prompt, reinforcement learning baked the habit of writing long chains into the model. The lever is identical — spend more compute by producing more tokens — but who pulls it moved from your prompt to a learned policy.

The consequences are direct. Appending "think step by step" to a reasoning model is usually pointless, and redundant instructions can distort or truncate the thinking; reach instead for the thinking-budget parameter each vendor exposes (names like reasoning effort or thinking budget). For ordinary models, CoT prompting remains live and useful, and "cheap model + CoT" versus "expensive reasoning model called plainly" is a per-task empirical question you should measure. The self-consistency idea survives too, as majority voting and verifier-based filtering inside reasoning models and in the orchestration layer above them.

How this gets used in practice

Who touches it, and when. Engineers shipping LLM features tuning a capability that isn't accurate enough — quote calculation, multi-condition ticket routing, aggregation over a table. Also evaluation owners deciding which model tier a task deserves, where "cheap model + CoT" is one of the options on the table.

Parameters you'll actually set. temperature (must be non-zero for self-consistency; start hunting around 0.5–0.9), n (sample count), max_tokens, stop sequences, the structured-output schema and its key order, and the thinking-budget parameter on reasoning models.

Pitfalls that turn into incidents.

• Hitting max_tokens. CoT lengthens the output, it clips right before the answer, the parser returns empty, and your eval counts a formatting failure as a reasoning failure. Raise the limit when you enable CoT and track the truncation rate as a metric.
• Running self-consistency at temperature 0. You get $N$ identical strings, every vote unanimous and worthless, and an $N$× bill for no accuracy.
• Answer extraction. "The answer is 5 cars", "5", and "5" will not all survive one regex. Specify the format explicitly, and if that still fails, fall back to the two-pass approach.
• Showing the chain-of-thought to end users. Since the written steps aren't guaranteed to be the actual basis of the answer, surfacing them as "the AI's reasoning" over-promises their reliability. Be especially careful in medical, financial, and hiring contexts.
• Logging cost. Storing the full chain for every request grows log volume by an order of magnitude. Decide what you keep before you turn it on.

A question that comes up in design reviews. "How is self-consistency different from beam search?" The core of the answer: self-consistency counts two differently-worded routes with the same final answer as the same vote — it selects for agreement on the answer, not for the highest-likelihood sequence. For how to validate prompting techniques generally, see the science of prompt engineering.

Summary

• CoT is a prompt that makes the model write intermediate steps — via few-shot exemplars, or a single zero-shot instruction
• It works because the written text re-enters the input, and because it converts fixed per-token depth into controllable output length
• Self-consistency runs several routes and votes on the final answer; it is dead at temperature 0 and costs linearly in the sample count
• Learn the failure conditions (small models, non-serial tasks) alongside the technique, and never mistake the written chain for the true cause of the answer
• Reasoning models are this same trick internalized through training — asking for it in the prompt is giving way to setting a budget for it