#kv-cache
6 articles
01
·Inference & Serving·★ MEMBER·PAPER·11 min read
Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as Well
The importance score that decides what leaves the KV cache turns out to buy almost nothing. Keep the prompt, evict uniformly at random inside each head, and you match the strongest prior evictor while serving 32-43% more tokens per second in vLLM. A walkthrough of the measurements across four models and six tasks, and the two controlled experiments that explain them.
02
·Inference & Serving·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Language Models Can Control Their Own Attention
During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot.
03
·★ MEMBER·PAPER·14 min read
Paper Walkthrough: Eleven Frames Are Enough — Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction
How do you track a camera and rebuild a scene from a video that never ends? ABot-Recon throws away long-range memory entirely and bets on the last twelve frames. A ground-up walkthrough of the paper.
04
·Inference & Serving·★ MEMBER·PAPER·11 min read
Surviving GPU Out-of-Memory — Every Cause, Every Fix
`CUDA out of memory` reports only the allocation that happened to fail last, which is almost never the culprit. We count what actually occupies VRAM in five buckets, derive the sixteen-bytes-per-parameter fixed cost of training, and work through the fixes in order of least damage: gradient checkpointing, optimizer compression, offloading, KV cache limits, and fragmentation.
05
·How Transformers Work·★ MEMBER·PAPER·9 min read
A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention
MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought.
06
·Inference & Serving·FREE·7 min read
The KV Cache from Scratch — The Heart of Fast Inference
An LLM emits one token at a time. Written naively, every single token costs a full recomputation of the whole sequence — a spectacular waste. The keys and values of past tokens never change again, and that one fact drops an entire order of magnitude. What you pay instead is memory, in an amount you can work out yourself, and that is why batch size and context length hit a ceiling.