transformer

Definition

An architecture that alternates self-attention with position-wise feed-forward layers, plus residual connections and normalization. Its decisive advantage over recurrent models is that attention relates all positions in one parallel operation, so training scales across accelerators and long-range dependencies are one hop away. Most current large language and vision models are variations on this template rather than departures from it.

Explainers using this term

• A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear AttentionFast Transformer Decoding: One Write-Head is All You Need

In the textbook

• Attention and the Transformer