#serving

6 articles

01 ·Time Series·★ MEMBER·PAPER·7 min read Paper Explained: STEPQuant — When and Where Quantization Errors Matter in Delta-Rule Recurrent States Linear-attention states have a fixed size, yet under concurrent serving they can outgrow the model weights. Naive low-bit quantization lets errors pile up with every update. STEPQuant spends bits and scales according to how long an error survives and which row it lands on. Explained from scratch. 02 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper walkthrough: BeaconKV — evicting KV cache by predicting when a reasoning model will re-read its own plan Deep into a long chain of thought, a reasoning model occasionally looks back at the plan it wrote at the start. The queries that do this cluster into a handful of groups in embedding space — so keeping a few dozen cluster representatives (beacons) is enough to predict which KV pairs will be revisited. A training-free method with up to 5.8× less memory and over 4.3× more throughput, read through its equations and code. 03 ·Inference & Serving·FREE·PAPER·9 min read KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context Inference Raise the context length and your concurrency collapses. The culprit is the KV cache, and its size is one plain multiplication. That formula has only three terms you can touch, so compression has only three directions — lower the precision, throw tokens away, share heads. KIVI, H2O, SnapKV, GQA and attention sinks all line up on those three. 04 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as Well The importance score that decides what leaves the KV cache turns out to buy almost nothing. Keep the prompt, evict uniformly at random inside each head, and you match the strongest prior evictor while serving 32-43% more tokens per second in vLLM. A walkthrough of the measurements across four models and six tasks, and the two controlled experiments that explain them. 05 ·Inference & Serving·FREE·PAPER·10 min read LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU Idle Getting a model to run and getting it to handle a hundred users are different problems. Without touching the weights or changing a single output token, the order and grouping of requests can multiply what one GPU delivers. This walks through why — arithmetic intensity, continuous batching, PagedAttention — and why throughput and latency can never both win. 06 ·Inference & Serving·FREE·7 min read The KV Cache from Scratch — The Heart of Fast Inference An LLM emits one token at a time. Written naively, every single token costs a full recomputation of the whole sequence — a spectacular waste. The keys and values of past tokens never change again, and that one fact drops an entire order of magnitude. What you pay instead is memory, in an amount you can work out yourself, and that is why batch size and context length hit a ceiling.