inference

Definition

Running a trained model forward to produce an output — the deployment-time counterpart to training. Aggregate inference cost typically dwarfs training cost over a model's lifetime, which is why latency, throughput, and quantisation get so much attention. Statisticians use "inference" for drawing conclusions about parameters from data; in ML engineering it almost always means execution.

Explainers using this term

• Paper walkthrough: BeaconKV — evicting KV cache by predicting when a reasoning model will re-read its own planBeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
• Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single OutputFast Inference from Transformers via Speculative Decoding

In the textbook

• When You Actually Use It — The Price of Inference