self-attention

Definition

The mechanism where every position in a sequence scores its relevance against every other position in the same sequence and aggregates their values accordingly. Each token projects to a query, a key, and a value; dot products of queries and keys set the weights. The consequence that matters is that any two positions are one hop apart regardless of distance, unlike recurrence, at a cost quadratic in sequence length.