#kv-cache

11 articles

01 ·★ MEMBER·PAPER·10 min read Paper Walkthrough: MaP-WAM — Turning a Robot's Memory into Plans So the Executor Stays the Same Size as History Grows For robot tasks you can't solve without remembering the past, MaP-WAM stops feeding the executor the full video history and instead compresses it into a plan for what to do next. The executor predicts its own progress alongside actions to decide when a step is done. We walk through the design, which reports 83.3% on RMBench and 78.0% on real robots, straight from the paper. 02 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: DeepSeek-V4.1-Flash — How Far Can KV Cache Compression Go? DeepSeek-V4.1-Flash shrinks its global KV cache to 890 bytes per token. We unpack the four ideas behind it (the CED layer split, cross-layer KV sharing in CSA2, FP4 caching, and SWA Bounded Replay) from first principles. 03 ·How Transformers Work·★ MEMBER·PAPER·15 min read Paper Walkthrough: SAS — Training a Sparse-Attention Block Selector End-to-End With the Language Modeling Loss The selector that decides which context blocks a query reads has traditionally been trained by imitating the dense model's attention distribution. SAS injects the selector's scores into the softmax as log-space gates, so the gradient that Top-K used to block flows straight from next-token prediction into the selector. We follow the paper through its four-axis ablation, the gradient formulas behind each choice, and the 10.6–15.5 point GPQA-Diamond gain at a 1024-token budget. 04 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper walkthrough: BeaconKV — evicting KV cache by predicting when a reasoning model will re-read its own plan Deep into a long chain of thought, a reasoning model occasionally looks back at the plan it wrote at the start. The queries that do this cluster into a handful of groups in embedding space — so keeping a few dozen cluster representatives (beacons) is enough to predict which KV pairs will be revisited. A training-free method with up to 5.8× less memory and over 4.3× more throughput, read through its equations and code. 05 ·Inference & Serving·FREE·PAPER·9 min read KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context Inference Raise the context length and your concurrency collapses. The culprit is the KV cache, and its size is one plain multiplication. That formula has only three terms you can touch, so compression has only three directions — lower the precision, throw tokens away, share heads. KIVI, H2O, SnapKV, GQA and attention sinks all line up on those three. 06 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as Well The importance score that decides what leaves the KV cache turns out to buy almost nothing. Keep the prompt, evict uniformly at random inside each head, and you match the strongest prior evictor while serving 32-43% more tokens per second in vLLM. A walkthrough of the measurements across four models and six tasks, and the two controlled experiments that explain them. 07 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: Language Models Can Control Their Own Attention During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot. 08 ·★ MEMBER·PAPER·14 min read Paper Walkthrough: Eleven Frames Are Enough — Revisiting Local Context for Long-Horizon Streaming 3D Reconstruction How do you track a camera and rebuild a scene from a video that never ends? ABot-Recon throws away long-range memory entirely and bets on the last twelve frames. A ground-up walkthrough of the paper. 09 ·Inference & Serving·FREE·PAPER·11 min read Surviving GPU Out-of-Memory — Every Cause, Every Fix `CUDA out of memory` reports only the allocation that happened to fail last, which is almost never the culprit. We count what actually occupies VRAM in five buckets, derive the sixteen-bytes-per-parameter fixed cost of training, and work through the fixes in order of least damage: gradient checkpointing, optimizer compression, offloading, KV cache limits, and fragmentation. 10 ·How Transformers Work·FREE·PAPER·9 min read A Field Guide to Attention Variants — MQA, GQA, Sliding Windows, Linear Attention MQA, GQA, sliding windows and linear attention are not four unrelated tricks. One multiplication decides how large a KV cache gets, and every variant is a decision about which factor in it to attack. Lined up against that formula, the family tree shows exactly what each one gave up and what it bought. 11 ·Inference & Serving·FREE·7 min read The KV Cache from Scratch — The Heart of Fast Inference An LLM emits one token at a time. Written naively, every single token costs a full recomputation of the whole sequence — a spectacular waste. The keys and values of past tokens never change again, and that one fact drops an entire order of magnitude. What you pay instead is memory, in an amount you can work out yourself, and that is why batch size and context length hit a ceiling.