attention

Definition

A mechanism that computes, for each position, a set of weights over other positions and returns their weighted sum — in practice a softmax over query-key dot products. It lets distant tokens interact directly in one step, which is why it displaced recurrence for long sequences. The name is a loose cognitive metaphor; the cost is quadratic in sequence length, driving a large literature on cheaper variants.

Explainers using this term

• Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as WellRandom Attention: Rethinking KV Cache Eviction for Efficient Reasoning
• Paper Walkthrough: Language Models Can Control Their Own AttentionLanguage Models Can Control Their Own Attention
• Video Understanding from Scratch — From a Pile of Frames to a Sense of TimeIs Space-Time Attention All You Need for Video Understanding? (TimeSformer)

In the textbook

• Attention and the Transformer