decoding

Definition

The procedure that turns a model's next-token distribution into actual text: greedy selection, beam search, temperature or nucleus sampling, and speculative variants that trade compute for latency. It happens at generation time with the weights untouched, which makes it one of the cheapest knobs available — quality and diversity often shift more with decoding settings than with fine-tuning.

Explainers using this term

• A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear AttentionFast Transformer Decoding: One Write-Head is All You Need
• Paper Walkthrough: Interpretable MEG Decoding of Perceived Speech — Reading the Decoder's Weights as a Brain MapInterpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrieval
• Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single OutputFast Inference from Transformers via Speculative Decoding