traffic

Definition

The actual flow of requests hitting a system. What matters is rarely the mean but the shape: diurnal cycles, bursts, and the joint distribution of arrival times with input and output lengths, all of which govern how well batching and scheduling work in an inference service. Measurements taken on real traffic carry considerably more weight than those from a synthetic load generator, which tends to be smoother than reality ever is.

Explainers using this term

• Paper Walkthrough: From Production Traffic to Post-Training — Folding 200 Internal Apps Into One Self-Hosted LLMFrom Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix