The Science of Prompt Engineering — What Is Proven and What Is Folklore

Does telling a model to think step by step actually work? Working only from the Chain-of-Thought paper (Wei et al., 2022): what the experiments genuinely established — an ability that emerges only above a scale threshold — and where the line falls between that and the incantations everyone repeats.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Incantations versus experiments

Prompt packs are for sale, everything opens with "you are a world-class expert", and word goes round that asking politely raises accuracy. Some of it may be true. The problem is that most of it has never been tested by anyone.

Some prompting techniques, though, have been properly measured, with the conditions pinned down. Chain-of-Thought (CoT) prompting is the canonical example. This article is not a CoT tutorial; its subject is drawing the line between what this paper established and what it did not touch.

What the CoT paper actually did

The intervention is startlingly plain. In each few-shot exemplar, alongside the answer, they wrote out the intermediate reasoning steps in natural language. Eight hand-written exemplars for the math word problems. That is all (§3.1). And the paper notes in parentheses that these exemplars did not undergo prompt engineering. The measurement is of the format, not of a polished incantation.

The effect is large. On GSM8K, PaLM 540B goes from 17.9% with standard prompting to 56.9%, and GPT-3 175B (text-davinci-002) from 15.6% to 46.9% (Appendix B, Tables 1 and 2). But it is not uniform. Harder problems gain more, and on SingleOp — the subset of MAWPS solvable in one step — the improvement is negative or very small (§3.2, Appendix Table 3). On the commonsense benchmark CSQA the gain was minimal, as the paper says itself (§4).

It only works when the model is big enough

The most reproducible finding here is not the effect but the condition on the effect.

Give the same prompts to smaller models and GSM8K accuracy goes like this (Appendix B, Table 2):

• LaMDA 420M: standard 2.6% → CoT 0.4%
• LaMDA 8B: 3.2% → 1.6%
• GPT 350M: 2.2% → 0.5%
• GPT 6.7B: 4.0% → 2.4%
• LaMDA 137B: 6.5% → 14.3%
• GPT 175B: 15.6% → 46.9%
• PaLM 540B: 17.9% → 56.9%

For small models it goes down. The paper's explanation is blunt: smaller models produced fluent but illogical chains of thought (§3.2). Appendix A.1 goes further — CoT actually hurts performance for most models below 10B.

Which is why the paper calls this an emergent ability: its success cannot be predicted by extrapolating small-scale models, and it appears only around the 100B parameter mark (§3.2, §8). The lesson is that "does this technique work?" is the wrong question shape. Whether it works is a function of model scale.

Three explanations the ablations killed

What makes this paper trustworthy is that it manufactures rival explanations for its own result and then kills them (§3.3). This is the heart of the matter.

Explanation 1: it works because it writes the equation. Drop the prose and have the model emit only a mathematical equation before answering. On GSM8K this barely helps (it does help on datasets of one or two steps). GSM8K questions are too semantically challenging to translate directly into an equation, is the paper's reading (Appendix A.4).

Explanation 2: it just spends more tokens, so more computation makes it smarter. To isolate that, they had the model emit only a run of dots (…) as long as the equation would have been. The result was about the same as baseline. Extra computation by itself is not the reason (§3.3).

Explanation 3: it merely activates relevant pretraining knowledge. So they put the chain of thought after the answer. Again about the same as baseline — meaning the final answer genuinely depends on the reasoning generated before it (§3.3).

All three fall, and what is left is exactly "write the intermediate steps, in natural language, before the answer, in order." That is the part that was demonstrated. Whether a claim comes with ablations of this kind is the difference between "it felt better" and "we know it works."

What is robust, what is brittle

How much does the wording matter? The paper answers in two halves (§3.4, Appendix A.2).

Robust — the direction. Chains of thought written independently by three co-authors all beat the baseline by a large margin; success does not depend on a particular linguistic style (§3.4). Reasoning written by crowd workers with no machine-learning background, taken straight from the GSM8K training set, performed comparably. Standard deviation across exemplar orderings is small in almost every case (Appendix A.2).

Brittle — the magnitude. The same paper states that prompt engineering still does matter. On the coin flip task, Annotator A reached 99.6% and Annotator C 71.4% — both above the 50.0% standard baseline, but 28 points apart (Appendix A.2, Table 7). And on a task of reversing a five-item list, two co-authors could not write a chain of thought that solved it despite their best attempts, while a third could (Appendix A.2).

So the line is: the direction is robust, the magnitude is not. Neither "wording is irrelevant" nor "the incantation is everything" is in this paper.

Temperature is one of the few knobs with a known mechanism

Unlike wording, sampling temperature has a mechanism you can write down: logits are divided by the temperature before the softmax, so a lower temperature sharpens the distribution onto the top token and a higher one flattens it.

But temperature is not a variable this paper tested. Every experiment used greedy decoding (§3.1). The only related thing the paper mentions is that follow-up work on self-consistency — sampling 21 chains at temperature 0.7 and taking the majority answer — improves on CoT (§3.1).

What remains untested

This paper ran one experiment: adding intermediate reasoning to the exemplars. Turn that around and the following are untested here:

• Asking politely, assigning a role ("you are an expert"), promising a reward or a penalty
• Capitalisation conventions, punctuation, choice of delimiters
• The zero-shot "let's think step by step" line. That is Kojima et al., a different paper, cited here only as related work (§7). It is routinely attributed to this paper, but these experiments are few-shot, with eight exemplars.

Transfer is not guaranteed either. The same prompts improved all three model families — LaMDA, GPT-3, PaLM — except on CSQA and StrategyQA for GPT-3. The paper names this a limitation and leaves the question of how pretraining data and architecture change the size of the gain to future work (Appendix A.2).

One practical principle drops out of that. A report that "X improved accuracy" is meaningless unless it comes with the model, the task, and the scale.

Without evaluation, prompt tuning reverts to folklore

What follows is a general point, not a claim of the paper. When you are judging by hand whether a prompt got better and you only have twenty examples, you may simply be selecting for those twenty. The more candidates you try, the further your local score drifts from performance on unseen inputs.

Limitations, in the paper's own words

• Whether it is "really reasoning" is unresolved. CoT emulates a human thought process, but the paper explicitly declines to say whether the network is reasoning (§6).
• No guarantee of a correct reasoning path. A generated chain can lead to right and wrong answers alike (§6). Among 50 correct answers examined, two arrived at the right answer coincidentally (§3.2).
• Annotation cost and serving cost. Minimal in the few-shot setting, potentially prohibitive for fine-tuning. And since it appears only at ~100B scale, it is costly to serve (§6).

How this shows up on the job

Prompting is now touched by every engineer shipping an LLM-backed feature and whoever owns output quality. There is an order to it: build the evaluation set first, then edit the wording. Reverse that and every subsequent judgement is a vibe.

The knob names. temperature, top_p, max_tokens, seed (for reproducibility), and response_format for structured output. Do not move temperature and top_p at the same time — that is settled practice. On the evaluation side, put something like promptfoo or DeepEval in place so before and after are compared on identical data.

Three traps. One: carrying CoT down to a small model. Table 2 says accuracy falls when the scale is not there, so re-validate the prompt whenever you switch to a cheaper model. Two: freezing prompts across a model upgrade — transfer is not guaranteed (Appendix A.2). Three: shipping the reasoning to production untouched. CoT multiplies output tokens, which lands directly on latency and the bill.

The design-review question is "will adding CoT improve accuracy?" Quote the paper's own conditions. CoT helps most when three things hold: the task is challenging and requires multi-step reasoning, a large model is used, and the scaling curve is relatively flat; the benefit shrinks as any of them fails (Appendix A.3). On one-step problems the improvement can even be negative. Start by asking whether your task meets those three conditions — that is the whole answer.

The idea of interleaving reasoning with tool use descends directly from here; see LLM agents from scratch. The temperature-and-softmax relationship itself is covered in probability for AI.

Summary

• What was demonstrated is "add intermediate reasoning to the exemplars". The rest of the folklore is untested here
• It works from roughly 100B parameters up; below that it lowers accuracy (LaMDA 420M: 2.6% → 0.4%)
• Not "more tokens" and not "knowledge retrieval" — both were killed by ablations (§3.3)
• The direction is robust (anyone's annotations win); the magnitude is not (99.6% vs 71.4% on the same task)
• Transfer across models is not guaranteed; GPT-3 on CSQA and StrategyQA is the counterexample (Appendix A.2)

Original paper: Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (arXiv:2201.11903, 2022)