A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention

MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought.

Fast Transformer Decoding: One Write-Head is All You Need

Attention is not one thing

Attention from scratch covered the textbook version: every token looks at every other token, dot products set the weights, and the values get mixed. It is a beautiful mechanism. Open the config file of a model that is actually in production, though, and the textbook form is the exception rather than the rule.

Keep the meeting metaphor going. You are the note-taker. You compare everyone's name tag (Key) against your own question (Query) and blend what people said (Value) into your notes. With ten people in the room this works fine. But a real LLM sits in a meeting where attendees keep arriving and nobody ever leaves. A context of 10,000 tokens means 10,000 name tags and 10,000 statements piled on the desk, none of them thrown away, every time you decide on the next word.

The variants are all answers to the question of how to fold that desk down. And the thing worth noticing is that they are not four unrelated inventions. There are only so many places you can fold, and each variant is named by the fold it chose.

Two different things are expensive

Start by separating the costs, because they get conflated constantly and they are not the same problem.

During training, and during the prefill stage where a whole prompt is read at once, the $n \times n$ score table itself is the burden ($n$ being the sequence length). Double the sequence and the table quadruples. That is a story about compute and memory.

During generation, when tokens come out one at a time, there is only ever one row of that table: the new token's Query against every past Key. And yet generation is slow. The reason is not arithmetic — it is that every past K and V has to be read back, in full, for every single token. That is the KV cache, and its size is decided entirely by a product.

In words: the cache size is layers × KV heads × head width × tokens accumulated × concurrent requests, multiplied by two (for K and V) and by the bytes per number. Reading it term by term: the leading $2$ covers K and V. $L$ is the number of layers, each holding its own cache. $h_{kv}$ is the number of KV heads. $d_{\mathrm{head}}$ is the width of one head. $n$ is how many tokens have piled up. $B$ is how many conversations you are serving at once. $p$ is bytes per number — 2 for FP16.

Everything here is multiplication, and that is the whole point. Cut any single factor to $1/8$ and the total drops to $1/8$. Which also means there is nowhere to save that is not already in this formula. So the family tree of variants sorts itself by which term got hit:

• shrink $h_{kv}$ → MQA / GQA (and MLA)
• shrink the effective $n$ → sliding window attention
• never build the $n \times n$ table at all → linear attention
• shrink $p$ → quantizing the KV cache

MQA — one shared roster

The first move was also the bluntest subtraction. Multi-Query Attention (MQA), proposed by Noam Shazeer in 2019, keeps all the Query heads but collapses K and V down to a single head. Everyone may ask their own question; everyone consults the same one roster, passed around the table. In the formula $h_{kv}$ becomes $1$, so the cache shrinks by a factor of the head count.

Why does that translate so directly into speed? Because generating one token means reading the entire KV cache back from memory. The arithmetic involved is trivial; memory bandwidth is what sets the pace. Cut the bytes read to $1/h$ and the wait shrinks with it. This is the explanation for the initially odd-sounding result that FLOPs barely change and yet the model gets faster.

The price is expressive power. The appeal of multiple heads was that each could look at something different — and MQA makes them all look at the same thing. The reported quality loss is small, but it is not zero.

GQA — split the difference

In 2023 Ainslie et al. published the compromise: Grouped-Query Attention (GQA). Divide the $h$ Query heads into $g$ groups and give each group its own shared K and V. Set $g = 1$ and you have MQA; set $g = h$ and you have ordinary multi-head attention. It is a continuous dial with both prior methods at its ends.

The other contribution mattered just as much in practice. The paper showed you can take an existing MHA checkpoint, initialise each group's KV head by averaging the heads in that group, and recover the quality with a small amount of extra training (uptraining) — negligible next to the original pretraining run. Not having to train from scratch is what made GQA spread. Llama 2 70B uses 64 Query heads against 8 KV heads, so its cache is one eighth of the MHA equivalent.

Sliding windows — cut how far you look

So far the question has been whose name tags do I keep. Now change the other one: how far back do I look. Under sliding window attention each token attends only to the previous $w$ tokens. Of the $n \times n$ table you keep a diagonal band of width $w$ and discard everything outside it. Cost becomes $n \times w$ — linear in sequence length, not quadratic.

The obvious objection is that the model can then never see the whole document. The answer is that stacking layers widens the receptive field. In layer one a token reaches $w$ back. In layer two the tokens it reaches have themselves already reached $w$ back, so information travels roughly $2w$. With $k$ layers the reach is on the order of $k \times w$ — a relay of whispers rather than a single long shout.

Longformer built this out for long documents in 2020, and Mistral 7B put it in a decoder with a 4096-token window in 2023, which is when most people met it. The payoff at generation time is the big one: only the most recent $w$ tokens need to stay cached, so memory stops growing with context length. The weakness is equally clear — tasks that need an exact quotation from outside the window are exactly what this breaks. Longformer patched around it by granting a few designated tokens global attention over the whole sequence.

Linear attention — drop the softmax, reassociate

The boldest variant is the one that refuses to build the $n \times n$ table in the first place.

Plain attention has to go through that table only because the softmax wraps all of $QK^\top$. Matrix multiplication is associative, so in principle $(QK^\top)V$ could be regrouped as $Q(K^\top V)$ — the first builds an $n \times n$ table, the second builds a small $d \times d$ box. A nonlinear softmax sitting in the middle is what forbids the regrouping.

So linear attention removes the softmax. Instead of exponentials, similarity is measured with dot products of a non-negative feature map $\phi$:

In words: the numerator holds a single box into which every past (name tag, statement) pair has been summed, and the current question is simply applied to that box. The denominator is the normaliser that keeps the weights summing to one. The load-bearing part is $\sum_{j \le i}$ — the past has been folded into one accumulator, and the size of that accumulator does not depend on how long the sequence is.

S = np.zeros((d_k, d_v))   # state: the name-tag x statement box
z = np.zeros(d_k)          # running sum for normalisation

for k, v, q in stream:               # one token at a time
    S += np.outer(phi(k), v)         # just add into the box
    z += phi(k)
    out = (phi(q) @ S) / (phi(q) @ z + 1e-6)

That loop is what Katharopoulos et al. meant by the title "Transformers are RNNs". At generation time linear attention is an RNN: one fixed-size state, updated per token. The KV cache stops being a list that grows with length and becomes a matrix of settled size.

So why is it not the default? Because what it gave up turned out to be expensive. Softmax can, when the situation calls for it, put essentially all of its weight on one token. Sharp reference — "this pronoun points at that noun three words back" — is built out of exactly that ability to commit. Dot products of non-negative feature maps have a hard time producing the same sharpness, and that limitation is widely held to be the main reason adoption in production models has stayed narrow.

Where things actually stand: blends

Real models do not pick one of these and discard the rest. The common shape today is GQA as the foundation, with sliding-window layers and full-attention layers alternating through the stack: the windowed layers read nearby context densely, and the occasional full layer picks up whatever is far away. There are also methods that compress K and V themselves into a low-rank latent vector before caching, such as MLA (Multi-head Latent Attention) in DeepSeek-V2.

One line worth drawing clearly: FlashAttention is not a variant. It processes the $n \times n$ table block by block inside fast on-chip memory so the full table never gets written out to slow memory. The output is mathematically identical to plain attention. It is not an approximation, so quality is unchanged — and it does not shrink the KV cache by a single byte.

How this is used on the job

The people who need this are the ones choosing models and the ones running inference infrastructure.

Choosing a model. Open config.json and compare num_attention_heads against num_key_value_heads. Equal means MHA; a KV count of 1 means MQA; anything between is GQA. Cache size is proportional to num_key_value_heads, so that one line roughly sets the order of magnitude of how many concurrent requests a given GPU will hold. If sliding_window is present, the model is windowed.

Standing up a server. On vLLM the handles that matter are --max-model-len, --gpu-memory-utilization and --kv-cache-dtype. Concurrency lands at roughly "memory available for KV cache ÷ cache size per request", so the multiplication from the top of this article is the estimate — you do not need anything more elaborate.

How it comes up in a design review. "Explain the difference between MQA and GQA, down to what actually gets faster." The chain to walk: you are cutting the number of KV heads → that cuts the bytes read back on every generated token → memory bandwidth is the bottleneck, so latency falls → and FLOPs are essentially unchanged. Landing that last clause is what makes the answer sound like experience.

Three ways to get hurt:

1. Tensor parallelism versus KV head count. A model with 8 KV heads does not divide across more than 8 GPUs. Some implementations paper over this by replicating KV heads, which quietly erodes the memory saving you were counting on. Check the KV head count before fixing the parallelism degree.
2. Reading "128k context" correctly. For a windowed model, the advertised length does not mean every token is equally reachable. If the job is pulling one exact fact out of a long document, verify on your own data before committing.
3. Expecting the wrong thing from FlashAttention. Enabling it will not reduce the KV cache. When you are hitting OOM, the levers that matter are usually context length, batch size and KV dtype instead.

Summary

• The variants organise cleanly by which factor of the KV-cache product they attack
• MQA and GQA cut $h_{kv}$; sliding windows cut the effective $n$; linear attention removes the $n \times n$ table entirely
• Generation is bound by memory bandwidth rather than compute, so cutting bytes read converts directly into speed
• FlashAttention is an exact implementation optimisation, not an approximation — and it leaves the KV cache untouched

The estimation procedure itself is written up in KV cache from scratch. Counting which term dominates your workload is the right first step, before picking a variant to adopt.