pagedattention

Definition

A KV-cache scheme that borrows paging from operating systems: instead of reserving one contiguous buffer per sequence, the cache is split into fixed-size blocks that need not be adjacent in memory. This removes the fragmentation and over-allocation that come from sizing buffers for the worst-case sequence length, so far more requests fit on the same GPU. It is the technique that anchors the vLLM serving engine, and blocks can also be shared across requests with a common prefix.

Explainers using this term

• LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU IdleEfficient Memory Management for Large Language Model Serving with PagedAttention