request

Definition

One incoming call to a model-serving system. Serving papers measure latency, throughput, and cost per request, and much of the engineering — continuous batching, KV-cache management, scheduling — exists because requests arrive at unpredictable times with wildly different prompt and output lengths. The term belongs to deployment rather than training.

Explainers using this term

• Paper Walkthrough: From Production Traffic to Post-Training — Folding 200 Internal Apps Into One Self-Hosted LLMFrom Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix