Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single Output

Speculative decoding lets a small model draft several tokens ahead while a large model verifies them in one batch — provably identical outputs, 2–3x faster in practice. A from-scratch walkthrough of the mechanism and the intuition behind the acceptance rate α, straight from the original paper.

Fast Inference from Transformers via Speculative Decoding

Why smarter models feel slower

Models like ChatGPT produce text one token (a fragment of a word) at a time. Each new token depends on everything written so far, so generating K tokens means running the model K times, strictly one after another. The paper points to this serial loop as the root cause of slow generation (Abstract).

There is a second, easily missed observation. According to the paper, inference from large models is often bottlenecked not on arithmetic but on memory bandwidth and communication (§1). Most of the time goes into streaming enormous weight matrices out of memory while the arithmetic units sit idle. The paper's starting point is to put that idle compute to work by increasing concurrency. For background on why memory dominates, see our article on the memory wall.

A metaphor: the junior writer and the chief editor

Having the chief editor (the large model) write every single word is slow. Instead, let a junior writer (a small model) draft a few words ahead; the editor then reads the whole draft at once and marks it up: "correct up to here — from this word on, I'd have written something else." One pass by the editor locks in several words at a time.

The idea comes from speculative execution in CPUs — branch prediction being the famous example: do work before you know it's needed, and win if it usually is. The paper generalizes this to the stochastic setting, where the "right" next task is only determined probabilistically (§1).

The big picture: draft → verify in one batch → patch (§2.1)

Call the model we want to accelerate the target model $M_p$, and the drafting helper the approximation model (draft model) $M_q$. One iteration works like this:

1. $M_q$ drafts $\gamma$ tokens the ordinary way (autoregressively)
2. $M_p$ is run once, in parallel, on the $\gamma+1$ prefixes — the original context, the context plus 1 drafted token, …, plus all $\gamma$ drafted tokens — yielding the target's probability distribution at every position
3. The drafted tokens are checked left to right with an acceptance rule (next section). The first rejected position is resampled from a corrected distribution; if everything passes, one bonus token is sampled on top

The design guarantees that each run of $M_p$ produces at least 1 and up to $\gamma+1$ tokens, so even in the worst case the number of serial target runs never exceeds plain autoregressive decoding (§2.1). In the paper's Figure 1, a 97M-parameter target, helped by a 6M-parameter draft model, generates a 38-token sentence in just 9 serial runs.

How often drafts "hit" also depends on the shape of the distribution. In the paper's measurements, sharper (lower-temperature) distributions gave higher acceptance rates (§4.2). You can get a feel for distribution sharpness below.

The heart of it: an acceptance rule that never changes the output distribution (§2.3)

"If a small model writes the drafts, won't quality drop?" — this is the paper's central contribution: a rule called speculative sampling guarantees that the output distribution is mathematically identical to what the target model would produce alone (§2.3, §A.1). A good draft model makes you faster; a bad one never makes the output worse — only slower. The rule that makes this almost-too-good deal work fits in three lines, and from here we'll walk through it, the proof that the distribution is preserved, and the acceptance rate $\alpha$ that governs the entire speedup.

For a drafted token $x$, write the target's probability as $p(x)$ and the draft model's as $q(x)$. Then:

• If $q(x) \leq p(x)$, accept unconditionally
• If $q(x) > p(x)$, reject with probability $1-\frac{p(x)}{q(x)}$
• On rejection, resample from the corrected distribution

$$p'(x) = \mathrm{norm}\big(\max(0,\; p(x) - q(x))\big)$$

In words: $p'(x)$ keeps only the tokens where the target assigns more probability than the draft model did, renormalized to sum to 1. Tokens the junior writer underrated are exactly the ones the resample favors. Read $p$ as how badly the chief editor wanted that word and $q$ as how badly the junior wanted it: the formula, in words, crosses out every word the junior over-claimed and redraws from whatever preference the editor has left over.

The proof that this preserves the distribution takes a few lines in §A.1. The probability that token $x$ ends up chosen is the sum of two routes:

• via acceptance: $q(x)\cdot\min(1, \frac{p(x)}{q(x)}) = \min(p(x), q(x))$
• via rejection and resampling: $p(x) - \min(p(x), q(x))$

Add them and you get exactly $p(x)$. Notably, the proof puts no conditions on $q$ whatsoever, so any draft model is fair game (§3.6). In numpy it is just:

def speculative_sample(p, q, x):        # x: the token M_q drafted
    if np.random.rand() < min(1.0, p[x] / q[x]):
        return x                        # accept
    resid = np.maximum(0.0, p - q)      # keep where the target bets more
    return sample(resid / resid.sum())  # resample from corrected dist.

The acceptance rate α decides everything (§3.1–3.2)

Call the probability that a draft passes the acceptance rate $\beta$, and its expectation $\alpha$. The paper proves a clean theorem: $\alpha$ is precisely the overlap between the two distributions (Theorem 3.5, Corollary 3.6).

$$\alpha = \sum_x \min(p(x),\, q(x))$$

Written out, that is a sum over every word the model might say next, which says: keep the smaller of the two probabilities each time, then add them all up. $\alpha$ is the share of opinion the junior writer and the chief editor already hold in common.

The more the draft model and the target put similar probability on the same tokens, the more drafts survive. Crucially, $\alpha$ is a property of the models and the task alone — you can measure it and estimate your speedup before touching any hardware.

Under the simplifying assumption that the $\beta$s are i.i.d., the expected number of tokens locked in per run of $M_p$ is (§3.1):

Read it as: how many tokens, on average, one target run advances the text. Spelled out, $\gamma$ is how many words the junior drafts before showing them and $\alpha$ is how often one of those words survives, so the fraction is simply "words gained per pass of the editor" — if every draft passed, a single pass would finish $\gamma+1$ words. It grows as $\alpha$ approaches 1 (drafts keep hitting) and as the draft length $\gamma$ grows. In the paper's Table 1, $\alpha=0.8$ with $\gamma=5$ gives a 3.69x speed factor when the draft's cost is negligible.

How much faster in practice? (§3.3, §4.1)

For wall time we must also pay for the drafting itself. Define the cost coefficient $c$ as the time of one draft-model run over one target run (in the paper's experiments, with drafts about two orders of magnitude smaller, $c$ was always below 0.05; §3.3). The expected wall-time improvement is

$$\frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c+1)}$$

The numerator is the tokens-per-run formula above; the $\gamma c+1$ in the denominator is the time for $\gamma$ draft runs plus one target run. The whole ratio therefore says, in words: words gained per round, divided by what the round cost you. Letting the junior draft longer lifts the top of the fraction, but it lifts $\gamma c$ at the bottom too — that tug-of-war is the entire tuning problem. Whenever $\alpha > c$ there exists a $\gamma$ that yields a speedup (Corollary 3.9), and the optimal $\gamma$ follows numerically from $\alpha$ and $c$ (§3.5).

Empirically, pairing T5-XXL (11B) with an off-the-shelf T5-small (77M) as the draft — no retraining, no architecture changes — gave measured speedups of 3.4x (argmax) / 2.6x (temperature 1) on English–German translation and 3.1x / 2.3x on summarization (§4.1, Table 2). Across other models, including LaMDA 137B, draft models roughly two orders of magnitude smaller produced $\alpha$ values between 0.5 and 0.9 (§4.2). At the extreme, even a plain bigram table as the draft achieved $\alpha\approx0.2$, good for a 1.25x speedup (§3.6).

No free lunch: total arithmetic goes up (§3.4)

One caveat. Running $M_p$ on $\gamma+1$ prefixes in parallel means the total number of arithmetic operations increases (Theorem 3.11) — rejected drafts are wasted compute. On the other hand, the number of memory reads for the weights and the KV cache shrinks by the tokens-per-run factor, since they are loaded once per iteration rather than once per token. That is why the paper positions the method for the typical LLM regime: arithmetic units idle, memory bandwidth saturated (§1, §6).

Limitations and what remains unverified

• It does not help when no spare compute is available — the paper says so explicitly (§6). In configurations already saturated on arithmetic, it can hurt.
• The formulas rest on the i.i.d. assumption for $\beta$, which is only an approximation; theoretical and measured speedups diverge somewhat (§A.3).
• Wall-time validation covers T5-XXL only, at batch size 1 on a single TPU-v4 (§4.1). For LaMDA and the GPT-like model, only $\alpha$ was measured.
• Compatibility with beam search is sketched with a performance penalty; its analysis is left as future work (§A.4).
• Only the text modality was tested; other domains such as images remain open (§6).
• An appendix explores relaxing the identical-distribution guarantee ("lenience") for extra speed, noting it may hurt sample diversity (§A.5). All headline results use the strict version.

How this is used in the field

The people who touch this are mostly inference-infrastructure engineers serving LLMs. It has become a standard lever when a chat or code-completion API must cut latency without buying more GPUs.

In practice it now ships inside the major serving stacks — --model-draft in llama.cpp, and vLLM's speculative-decoding settings, where num_speculative_tokens corresponds to the paper's $\gamma$. The paper's theory transfers directly to tuning: the one knob that matters is $\gamma$, and pushing it too high inflates wasted compute on rejections (the trade-off in Table 1). Since $\alpha$ depends on the task and the temperature, measure it on your own traffic before settling on a value.

Three pitfalls cause real incidents. First, on servers already compute-saturated with large batches, the method doesn't help and can make things worse — exactly the limitation in §6; the sweet spot is low-batch, latency-sensitive serving. Second, in practice the draft and target must share a tokenizer (vocabulary), or the drafted tokens can't be compared position by position; the paper's experiments likewise use one tokenizer across all models (§4.2). Third, high-temperature workloads lower $\alpha$ and shrink the gains — Table 2 shows the gap between temperature 0 and 1.

If an interviewer asks "why does it get faster without changing the output?", the two-line answer from §A.1 is all you need: acceptance contributes $\min(p,q)$, resampling contributes $p-\min(p,q)$, and they sum back to $p$.

Wrapping up

Speculative decoding is disarmingly simple — a small model drafts, a large model verifies in one batch — and the arithmetic of the acceptance rule guarantees exactly identical outputs. The speed comes from the acceptance rate $\alpha$, the overlap between the two distributions, and off-the-shelf model pairs already yield measured 2–3x gains. For other routes to faster inference see quantization explained, and for the memory-bound backdrop, the memory wall and the KV cache.