KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context Inference

Raise the context length and your concurrency collapses. The culprit is the KV cache, and its size is one plain multiplication. That formula has only three terms you can touch, so compression has only three directions — lower the precision, throw tokens away, share heads. KIVI, H2O, SnapKV, GQA and attention sinks all line up on those three.

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

"I raised the context length and my concurrency collapsed"

Start serving an LLM yourself and this is usually the first wall you hit. The weights fit on the GPU with room to spare. Then you bump the context length from 4K to 32K, and the number of requests you can run in parallel falls off a cliff — sometimes far enough that a single request out-of-memories.

The weights did not grow by a single byte. What grew is the KV cache: the memory held by the trick of stashing past keys and values so you never recompute them. KV Caching from Scratch covers the mechanism, but the takeaway fits in one line. The cache grows linearly in both context length and request count.

This article is about cutting it back down. KIVI, H2O, SnapKV, GQA, attention sinks — the names are scattered, but there were only ever three places to cut, and every method is hitting one of them.

An analogy: carrying the whole meeting transcript

A KV cache is like the transcript of a long meeting. Before you say anything new, you reread every earlier remark — so the transcript can never be thrown out, and the longer the meeting runs, the thicker the stack gets and the more of your desk (GPU memory) it occupies.

There are exactly three sensible ways to thin a stack of paper: write smaller = quantization, throw pages out = eviction, stop writing the same thing several times = sharing. Those three map one-to-one onto KV cache compression, and the correspondence is that clean because of the formula below.

Why KV eats memory — one multiplication

The size of the cache isn't an estimate or a theory. It's a shopping list.

In words, the cache holds two vectors (K and V) for every layer, for every token, across the KV heads, at the chosen precision, for every request in flight. Nothing in the formula is clever: it is just a bill of materials. And only three of its terms are yours to move at inference time: $b$, the number of tokens you actually retain out of $n$, and $H_{kv}$.

Symbol by symbol. The leading $2$ is for K and V, two of them. $L$ is the number of layers (each keeps its own K and V). $n$ is the context length — how many tokens you're currently holding. $H_{kv}$ is the number of KV heads and $d_h$ the dimension of one head, so their product is the size of one key, per layer, per token. $b$ is bytes per element (2 for fp16). $B$ is how many requests are in flight.

So it says only this: for every layer, for every token, hold two vectors, across the heads, at the chosen precision, times the number of users — not one term of it is clever. And what matters is that only three of those terms are yours to move. $L$ and $d_h$ are the model's architecture; at inference time you don't get a vote. $n$ is the context the user sent and $B$ is the concurrency you're trying to serve — shrinking those is losing, not winning. What's left is $b$ (precision), the number of tokens you actually retain out of $n$, and $H_{kv}$ (head count). Exactly the three moves from the paper-stack analogy.

Direction 1: lower the precision — K and V want different treatment

The most direct move is to attack $b$. A cache held in fp16 (2 bytes) can be held in int8 (1 byte) or int4 (half a byte), halving or quartering it outright. The mechanics are the same as in the quantization article: express the value range with a scale and a zero point, then round onto an integer grid.

But KV caches carry a wrinkle that weight quantization doesn't: keys and values are distributed differently. That's the observation KIVI (2024) builds on. The key tensor has a strong per-channel skew — a few specific channels (dimensions) run far larger than the rest, and it's the same channels regardless of which token you look at. The value tensor shows no comparable consistent structure.

That asymmetry decides how you group values for quantization. Rounding error spreads across whichever group shares a scale, so a single outlier poisons everything grouped with it. Hence: quantize keys along the channel axis, keeping outlier channels out of everyone else's group; quantize values along the token axis. This asymmetric pairing is what lets KIVI push to 2 bits with no fine-tuning — the "asymmetric" in the title points at exactly this. In practice it's paired with a residual window that keeps the most recent few dozen to few hundred tokens at full precision.

Direction 2: throw tokens away — attention is already lopsided

Next, reduce the number of tokens you retain at all. "Drop the oldest first" isn't good enough on its own, though, and it's worth being precise about why dropping works.

Attention weights pass through a softmax, which exponentially amplifies differences in score. A modestly higher score takes most of the mass and the rest collapses toward zero. Inspect a real model's attention matrices and you see it: a small handful of tokens absorbs most of the attention mass.

H2O (Heavy-Hitter Oracle, 2023) uses that skew directly. It accumulates how much attention weight each token has received so far, keeps the top "heavy hitters" plus the most recent tokens, and evicts the rest. Fix a cache budget and memory stays flat no matter how far the context runs.

SnapKV (2024) aims at a different moment. Where H2O prunes as it generates, SnapKV prunes in one shot, the instant the prompt has been read. It takes the last few dozen tokens of the prompt as an observation window, tallies which earlier positions that window attends to, and keeps the positions that won the vote. It then applies pooling so neighboring positions get selected together rather than slicing a sentence into fragments — which suits workloads that throw long prompts repeatedly.

What both share is that evicted KV never comes back. Unlike quantization, which merely coarsens, eviction is irreversible — operationally a much heavier difference than it sounds.

The trap: never evict the first tokens

Implement eviction yourself and you'll likely walk into the same trap everyone does. You reason "just keep the last N," write a sliding window, and the output falls apart the moment the text outgrows the window.

The cause is a phenomenon called the attention sink. As the StreamingLLM paper (2023) pointed out, trained Transformers pour a large amount of attention onto the first few tokens of the sequence for reasons that have nothing to do with meaning. Softmax forces the weights to sum to one, so even when a head has nothing it wants to look at, that mass has to go somewhere — and the position visible to every token, the very beginning, becomes the dump.

So when a sliding window evicts the beginning, the model's homeless attention gets forced onto genuinely meaningful tokens and the distribution breaks. The fix is almost anticlimactically simple: always retain the first few tokens (the paper reports about four is enough) and slide the window over everything after them. This "sink plus window" arrangement is now the standard shape for streaming long-context inference.

The lesson generalizes. An eviction policy can collide with a habit the model acquired during training — and the rule that looks most evenhanded is often the one that breaks things.

Direction 3: share — fewer KV heads

That leaves $H_{kv}$. This one isn't an inference-time trick at all; it's a model design decision.

Plain multi-head attention with 32 query heads also keeps 32 key and value heads. MQA asked whether all heads could share a single set of keys and values; that cost too much quality, so GQA (2023) split the difference, sharing K and V across groups of query heads — say 8 instead of 32. Take $H_{kv}$ from 32 down to 8 and the cache is a quarter of the size. Because it hits a term in the formula directly, there's no uncertainty about whether it works.

Some designs push further and store K and V as a low-dimensional latent that gets expanded on use (DeepSeek-V2's Multi-head Latent Attention, for instance); the field guide to attention variants lays out that lineage. If you're not training your own model, this direction is something you choose, not something you tune — which is the point. Checking $H_{kv}$ during model selection determines how much pain you'll have later with quantization and eviction.

In code: a minimal sink-plus-window

Eviction, stripped down, is just indexing into the cache.

def evict(cache_k, cache_v, n_sink=4, n_window=1020):
    # always retain the first n_sink tokens (the attention sink)
    if cache_k.shape[0] <= n_sink + n_window:
        return cache_k, cache_v
    keep = list(range(n_sink)) + list(range(cache_k.shape[0] - n_window, cache_k.shape[0]))
    return cache_k[keep], cache_v[keep]

Set n_sink to 0 and this is an ordinary sliding window. That one-line difference is what decides whether long generations survive. Note too that whether retained positions get renumbered contiguously or keep their original indices changes the output.

How this shows up on the job

KV cache compression is work for whoever serves the model: the person sizing the GPU order, the person committing to a concurrency SLO, the person who gets the ticket that says "it dies on long documents."

Start with precision and quantity. In vLLM: --kv-cache-dtype fp8 (low precision for the cache only), --max-model-len (context ceiling), --max-num-seqs (concurrency), --enable-prefix-caching (reuse of shared prefixes). The order matters: drop $b$ first, move to a model with smaller $H_{kv}$ second, reach for eviction last. The first two have predictable effects; eviction's failure mode depends on the input.

Evaluate on the task that compression hurts most. Perplexity on short completions, and chat that merely looks fine, barely degrade under eviction. What breaks is retrieving one specific fact from a long context — QA over long documents, references pulled from a codebase, needle-in-a-haystack tests. "It was fine in normal chat so we shipped it," followed by a long-context request in production, is the single most common way this goes wrong.

Keep track of what's reversible. Quantization only coarsens; eviction deletes. If your product leans on multi-turn conversations that refer back to a proper noun from several turns ago, treat eviction carefully. A one-shot summarization workload, by contrast, is a natural fit for prefill-time compression like SnapKV.

In design reviews the question is "what doubles when you double the context?" KV memory doubles (linear); prefill attention quadruples (quadratic). And whether you can explain why a window without a sink breaks is what separates real understanding from name recognition.

Summary

• Cache size is $2 L n H_{kv} d_h b B$. At inference time you can only move three things: $b$, the number of tokens retained, and $H_{kv}$
• Lower precision: KIVI exploits the asymmetry that keys have consistent outlier channels and values don't — group keys per-channel, values per-token
• Evict: H2O keeps heavy hitters by accumulated attention; SnapKV prunes in one shot at prefill using votes from an observation window
• Never drop the head of the sequence: models use the first few tokens as an attention sink. Sink-plus-window is the standard fix
• Share: GQA reduces $H_{kv}$ directly — a model-selection decision, not an inference-time knob
• Quantization coarsens; eviction destroys. Evaluate with long-context retrieval tasks