write-head

Definition

The key/value side of attention, as opposed to the query side. The term is inseparable from the multi-query attention result that one write-head suffices: keep many query heads but share a single key and value projection, and the KV cache shrinks by a factor of the head count — the thing that actually limits memory-bandwidth-bound decoding. It was later softened into grouped-query attention, which shares keys and values across small groups of heads.

Explainers using this term

• A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear AttentionFast Transformer Decoding: One Write-Head is All You Need