serving

Definition

The engineering of running a trained model in production: batching incoming requests, managing the KV cache, packing GPU memory, and keeping tail latency acceptable. It is judged on throughput and latency per unit of hardware, not on accuracy — the weights are fixed. Distinct from training, which optimises parameters rather than request handling.

Explainers using this term

• LLM Serving from Scratch — vLLM, Continuous Batching, and Not Letting the GPU IdleEfficient Memory Management for Large Language Model Serving with PagedAttention
• Paper Explained: FreeToken — Treating Your Own PC as a Single Elastic Inference PlatformFreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
• Paper Walkthrough: CodeNib — A Multi-View Data System That Serves Repository Context to Coding AgentsCodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents