quantization

Definition

Representing weights and activations with fewer bits — 8-bit or 4-bit integers instead of 16-bit floats — to cut memory and bandwidth. Since most inference is memory-bound, this usually buys real speedups as well as a smaller footprint, and the whole research problem is bounding the accuracy loss. Unlike pruning or distillation, it leaves the architecture untouched and only coarsens how numbers are stored.

Explainers using this term

• Paper Explained: Why Gated DeltaNet Survives 4-Bit Quantization — NVFP4 W4A4 in a Hybrid 27BWhy Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM