inference

Definition

Running a trained model forward to produce an output — the deployment-time counterpart to training. Aggregate inference cost typically dwarfs training cost over a model's lifetime, which is why latency, throughput, and quantisation get so much attention. Statisticians use "inference" for drawing conclusions about parameters from data; in ML engineering it almost always means execution.

Explainers using this term

• Speculative Decoding from Scratch — How a Tiny Draft Model Speeds Up an LLM Without Changing a Single OutputFast Inference from Transformers via Speculative Decoding