LLM Quantization from Scratch — Why Losing Precision Doesn't Break It

Cut a 16-bit weight down to 4 bits and the model keeps writing. Why doesn't it break? Starting from two numbers — scale and zero point — through the reason weights survive but activations resist (outliers), the fork between PTQ and QAT, what INT8 and INT4 actually do, and the single most common evaluation accident: signing off on perplexity alone.

Quantization is rounding onto a scale

Step on a bathroom scale and it reads "63.5 kg". Your actual mass might be 63.4827… kg, but the display is rounded onto 0.1 kg gradations. Make the gradations coarser and you need fewer digits to record the number — and the gap from the true value widens.

That is all quantization is. Map continuously scattered values onto a finite set of grid points, and cut the number of bits needed per value.

A language model's weight is normally held in fp16 or bf16 — two bytes. Hold it in INT4 (half a byte) and the model's memory footprint drops to a quarter. Memory is not the only thing that improves. As the KV cache article showed, the decode phase is memory-bandwidth bound: the time to read the whole weight set every step dominates. Reading a quarter as many bytes converts directly into speed.

And to clear up a misconception: it is not that "fp16 is continuous and INT4 is discrete". Floating point can only represent finitely many values too, so it is already quantized. The only difference is how the grid is laid out — floating point uses an exponential grid, fine near zero and coarse far from it, while integer quantization uses a uniform one.

You have already seen this exact operation

Dividing by a step size and rounding sits at the heart of image compression too. JPEG rewrites 8×8 blocks as frequency components, then divides each component by an entry of the quantization table and rounds — fine steps for low frequencies, coarse ones for high. The entire reason JPEG is lossy is contained in that one round.

What differs is the justification for what may be thrown away. JPEG could pick its targets by frequency, on the physiological grounds that the eye is blunt about fine detail. An LLM offers no such handle, which is exactly why deciding what to protect and what to round coarsely is where the engineering lives.

Scale and zero point

(The internals of the formats themselves — FP32, BF16, FP8 — are in Number formats: from FP32 to FP8 and INT4. This article assumes them and covers the procedure.)

Two numbers have to be fixed before rounding: the scale $s$, which maps a range of reals onto a range of integers, and the zero point $z$, which decides which integer real zero lands on. Quantizing to $b$ bits,

where $x_{\max}$ and $x_{\min}$ are the largest and smallest values being quantized. Put in words: the spread of the real values, divided by how many integers you can afford, is the width of one step — and the zero point is just how many of those steps sit between the smallest value and real zero. Even at 8 bits there are only 256 integers, so the wider the spread, the coarser the steps.

Quantizing and dequantizing then look like this.

Put in words: divide the value by the step width, round to the nearest whole number, and shove anything that falls off the end back inside the range the bits can hold; to read it back, multiply by the step width again. $q$ is the integer that gets stored and $\hat{x}$ the real that comes back. They do not agree; the gap is at most half a step, $s/2$. The one and only place information is destroyed is that round — precisely the situation in JPEG.

import numpy as np

def quantize(x, bits=8):
    qmax = 2 ** bits - 1
    s = (x.max() - x.min()) / qmax          # the step size
    z = round(-x.min() / s)                 # the grid point real zero lands on
    q = np.clip(np.round(x / s) + z, 0, qmax).astype(np.uint8)
    return q, s, z

def dequantize(q, s, z):
    return s * (q.astype(np.float32) - z)   # the rounded-off fraction is gone