transformer

Definition

An architecture that alternates self-attention with position-wise feed-forward layers, plus residual connections and normalization. Its decisive advantage over recurrent models is that attention relates all positions in one parallel operation, so training scales across accelerators and long-range dependencies are one hop away. Most current large language and vision models are variations on this template rather than departures from it.

Explainers using this term

• Paper Deep Dive: Your Transformer Can Hold Two Thoughts at Once — Blend Two Texts and the LLM Still Remembers BothYour Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
• Paper Explained: FreeFlow — Optical Flow with a Hierarchical Transformer and No Flow-Specific PartsFreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation
• A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear AttentionFast Transformer Decoding: One Write-Head is All You Need

In the textbook

• Attention and the Transformer
• Splitting Heads, Assembling Blocks — The Whole Transformer