low-precision

Definition

Representing and computing with fewer bits — FP16, BF16, FP8, INT8, INT4 — to cut memory footprint and bandwidth and to use hardware units built for narrow formats. The difficulty is dynamic range: with few exponent or mantissa bits, activation outliers overflow or get crushed, so scaling factors, per-channel granularity, and outlier handling do most of the real work. Quantization is the act of mapping into such a format; low-precision describes the regime you end up in.