quantization

Definition

Representing weights and activations with fewer bits — 8-bit or 4-bit integers instead of 16-bit floats — to cut memory and bandwidth. Since most inference is memory-bound, this usually buys real speedups as well as a smaller footprint, and the whole research problem is bounding the accuracy loss. Unlike pruning or distillation, it leaves the architecture untouched and only coarsens how numbers are stored.

Explainers using this term

• Paper Explained: STEPQuant — When and Where Quantization Errors Matter in Delta-Rule Recurrent StatesSTEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
• Paper Explained: Disaggregated Quantization — Quantize LLM Prefill and Decode DifferentlyDisaggregated Quantization: Specializing LLM Prefill and Decode
• KV Cache Compression — Quantize, Evict, Share Your Way to Cheaper Long-Context InferenceKIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache