#serving
3 articles
01
·Inference & Serving·★ MEMBER·PAPER·11 min read
Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as Well
The importance score that decides what leaves the KV cache turns out to buy almost nothing. Keep the prompt, evict uniformly at random inside each head, and you match the strongest prior evictor while serving 32-43% more tokens per second in vLLM. A walkthrough of the measurements across four models and six tasks, and the two controlled experiments that explain them.
02
·Inference & Serving·FREE·PAPER·10 min read
LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU Idle
Getting a model to run and getting it to handle a hundred users are different problems. Without touching the weights or changing a single output token, the order and grouping of requests can multiply what one GPU delivers. This walks through why — arithmetic intensity, continuous batching, PagedAttention — and why throughput and latency can never both win.
03
·Inference & Serving·FREE·7 min read
The KV Cache from Scratch — The Heart of Fast Inference
An LLM emits one token at a time. Written naively, every single token costs a full recomputation of the whole sequence — a spectacular waste. The keys and values of past tokens never change again, and that one fact drops an entire order of magnitude. What you pay instead is memory, in an amount you can work out yourself, and that is why batch size and context length hit a ceiling.