#vllm
3 articles
01
·Inference & Serving·★ MEMBER·PAPER·11 min read
Paper Walkthrough: Random Attention — Throwing KV Cache Entries Away at Random Works Just as Well
The importance score that decides what leaves the KV cache turns out to buy almost nothing. Keep the prompt, evict uniformly at random inside each head, and you match the strongest prior evictor while serving 32-43% more tokens per second in vLLM. A walkthrough of the measurements across four models and six tasks, and the two controlled experiments that explain them.
02
·Inference & Serving·★ MEMBER·PAPER·9 min read
Paper Walkthrough: Language Models Can Control Their Own Attention
During long-context decoding, a model re-reads its entire KV cache at every step. Declarative Attention has the model announce, inside its own chain-of-thought, where it will look next, and the inference engine builds an attention mask from that text. A walkthrough of the paper that cut attended tokens by 52.0% and 31.1% on off-the-shelf models, zero-shot.
03
·Inference & Serving·FREE·PAPER·10 min read
LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU Idle
Getting a model to run and getting it to handle a hundred users are different problems. Without touching the weights or changing a single output token, the order and grouping of requests can multiply what one GPU delivers. This walks through why — arithmetic intensity, continuous batching, PagedAttention — and why throughput and latency can never both win.