speculative

Definition

A pattern where a cheap process guesses ahead and an expensive one verifies, so you profit whenever the guess is right. The canonical case is speculative decoding: a small draft model proposes several tokens, the large model checks them in one forward pass, and generation restarts from the first rejection. Verification preserves the target model's output distribution, so the speedup costs no quality — it comes from better use of memory bandwidth, not from approximation.

Explainers using this term

• CPU Pipelines and Branch Prediction — The Factory Inside One Clock TickSpectre Attacks: Exploiting Speculative Execution
• Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single OutputFast Inference from Transformers via Speculative Decoding