Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single Output

Speculative decoding lets a small model draft several tokens ahead while a large model verifies them in one batch — provably identical outputs, 2–3x faster in practice. A from-scratch walkthrough of the mechanism and the intuition behind the acceptance rate α, straight from the original paper.

Fast Inference from Transformers via Speculative Decoding

Why smarter models feel slower

Models like ChatGPT produce text one token (a fragment of a word) at a time. Each new token depends on everything written so far, so generating K tokens means running the model K times, strictly one after another. The paper points to this serial loop as the root cause of slow generation (Abstract).

There is a second, easily missed observation. According to the paper, inference from large models is often bottlenecked not on arithmetic but on memory bandwidth and communication (§1). Most of the time goes into streaming enormous weight matrices out of memory while the arithmetic units sit idle. The paper's starting point is to put that idle compute to work by increasing concurrency. For background on why memory dominates, see our article on the memory wall.

A metaphor: the junior writer and the chief editor

Having the chief editor (the large model) write every single word is slow. Instead, let a junior writer (a small model) draft a few words ahead; the editor then reads the whole draft at once and marks it up: "correct up to here — from this word on, I'd have written something else." One pass by the editor locks in several words at a time.

The idea comes from speculative execution in CPUs — branch prediction being the famous example: do work before you know it's needed, and win if it usually is. The paper generalizes this to the stochastic setting, where the "right" next task is only determined probabilistically (§1).

The big picture: draft → verify in one batch → patch (§2.1)

Call the model we want to accelerate the target model $M_p$, and the drafting helper the approximation model (draft model) $M_q$. One iteration works like this:

1. $M_q$ drafts $\gamma$ tokens the ordinary way (autoregressively)
2. $M_p$ is run once, in parallel, on the $\gamma+1$ prefixes — the original context, the context plus 1 drafted token, …, plus all $\gamma$ drafted tokens — yielding the target's probability distribution at every position
3. The drafted tokens are checked left to right with an acceptance rule (next section). The first rejected position is resampled from a corrected distribution; if everything passes, one bonus token is sampled on top

The design guarantees that each run of $M_p$ produces at least 1 and up to $\gamma+1$ tokens, so even in the worst case the number of serial target runs never exceeds plain autoregressive decoding (§2.1). In the paper's Figure 1, a 97M-parameter target, helped by a 6M-parameter draft model, generates a 38-token sentence in just 9 serial runs.

How often drafts "hit" also depends on the shape of the distribution. In the paper's measurements, sharper (lower-temperature) distributions gave higher acceptance rates (§4.2). You can get a feel for distribution sharpness below.

The heart of it: an acceptance rule that never changes the output distribution (§2.3)

"If a small model writes the drafts, won't quality drop?" — this is the paper's central contribution: a rule called speculative sampling guarantees that the output distribution is mathematically identical to what the target model would produce alone (§2.3, §A.1). A good draft model makes you faster; a bad one never makes the output worse — only slower. The rule that makes this almost-too-good deal work fits in three lines, and from here we'll walk through it, the proof that the distribution is preserved, and the acceptance rate $\alpha$ that governs the entire speedup.