A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention

MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought.

Fast Transformer Decoding: One Write-Head is All You Need

Attention is not one thing

Attention from scratch covered the textbook version: every token looks at every other token, dot products set the weights, and the values get mixed. It is a beautiful mechanism. Open the config file of a model that is actually in production, though, and the textbook form is the exception rather than the rule.

Keep the meeting metaphor going. You are the note-taker. You compare everyone's name tag (Key) against your own question (Query) and blend what people said (Value) into your notes. With ten people in the room this works fine. But a real LLM sits in a meeting where attendees keep arriving and nobody ever leaves. A context of 10,000 tokens means 10,000 name tags and 10,000 statements piled on the desk, none of them thrown away, every time you decide on the next word.

The variants are all answers to the question of how to fold that desk down. And the thing worth noticing is that they are not four unrelated inventions. There are only so many places you can fold, and each variant is named by the fold it chose.

Two different things are expensive

Start by separating the costs, because they get conflated constantly and they are not the same problem.

During training, and during the prefill stage where a whole prompt is read at once, the $n \times n$ score table itself is the burden ($n$ being the sequence length). Double the sequence and the table quadruples. That is a story about compute and memory.

During generation, when tokens come out one at a time, there is only ever one row of that table: the new token's Query against every past Key. And yet generation is slow. The reason is not arithmetic — it is that every past K and V has to be read back, in full, for every single token. That is the KV cache, and its size is decided entirely by a product.

In words: the cache size is layers × KV heads × head width × tokens accumulated × concurrent requests, multiplied by two (for K and V) and by the bytes per number. Reading it term by term: the leading $2$ covers K and V. $L$ is the number of layers, each holding its own cache. $h_{kv}$ is the number of KV heads. $d_{\mathrm{head}}$ is the width of one head. $n$ is how many tokens have piled up. $B$ is how many conversations you are serving at once. $p$ is bytes per number — 2 for FP16.

Everything here is multiplication, and that is the whole point. Cut any single factor to $1/8$ and the total drops to $1/8$. Which also means there is nowhere to save that is not already in this formula. So the family tree of variants sorts itself by which term got hit:

• shrink $h_{kv}$ → MQA / GQA (and MLA)
• shrink the effective $n$ → sliding window attention
• never build the $n \times n$ table at all → linear attention
• shrink $p$ → quantizing the KV cache

MQA — one shared roster

The first move was also the bluntest subtraction. Multi-Query Attention (MQA), proposed by Noam Shazeer in 2019, keeps all the Query heads but collapses K and V down to a single head. Everyone may ask their own question; everyone consults the same one roster, passed around the table. In the formula $h_{kv}$ becomes $1$, so the cache shrinks by a factor of the head count.

Why does that translate so directly into speed? Because generating one token means reading the entire KV cache back from memory. The arithmetic involved is trivial; memory bandwidth is what sets the pace. Cut the bytes read to $1/h$ and the wait shrinks with it. This is the explanation for the initially odd-sounding result that FLOPs barely change and yet the model gets faster.

The price is expressive power. The appeal of multiple heads was that each could look at something different — and MQA makes them all look at the same thing. The reported quality loss is small, but it is not zero.