management

Definition

Allocating and reclaiming a scarce resource. In serving systems it means KV-cache and GPU memory management; in agent systems it means context management — deciding what stays in the window and what gets summarized or dropped. The word signals that a paper's contribution lies in systems design rather than in the model itself, which is often where the larger practical wins are.

Explainers using this term

• LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU IdleEfficient Memory Management for Large Language Model Serving with PagedAttention