LLM Quantization from Scratch — Why Losing Precision Doesn't Break It

Cut a 16-bit weight down to 4 bits and the model keeps writing. Why doesn't it break? Starting from two numbers — scale and zero point — through the reason weights survive but activations resist (outliers), the fork between PTQ and QAT, what INT8 and INT4 actually do, and the single most common evaluation accident: signing off on perplexity alone.

Quantization is rounding onto a scale

Step on a bathroom scale and it reads "63.5 kg". Your actual mass might be 63.4827… kg, but the display is rounded onto 0.1 kg gradations. Make the gradations coarser and you need fewer digits to record the number — and the gap from the true value widens.

That is all quantization is. Map continuously scattered values onto a finite set of grid points, and cut the number of bits needed per value.

A language model's weight is normally held in fp16 or bf16 — two bytes. Hold it in INT4 (half a byte) and the model's memory footprint drops to a quarter. Memory is not the only thing that improves. As the KV cache article showed, the decode phase is memory-bandwidth bound: the time to read the whole weight set every step dominates. Reading a quarter as many bytes converts directly into speed.

And to clear up a misconception: it is not that "fp16 is continuous and INT4 is discrete". Floating point can only represent finitely many values too, so it is already quantized. The only difference is how the grid is laid out — floating point uses an exponential grid, fine near zero and coarse far from it, while integer quantization uses a uniform one.

You have already seen this exact operation

Dividing by a step size and rounding sits at the heart of image compression too. JPEG rewrites 8×8 blocks as frequency components, then divides each component by an entry of the quantization table and rounds — fine steps for low frequencies, coarse ones for high. The entire reason JPEG is lossy is contained in that one round.

What differs is the justification for what may be thrown away. JPEG could pick its targets by frequency, on the physiological grounds that the eye is blunt about fine detail. An LLM offers no such handle, which is exactly why deciding what to protect and what to round coarsely is where the engineering lives.

Scale and zero point

(The internals of the formats themselves — FP32, BF16, FP8 — are in Number formats: from FP32 to FP8 and INT4. This article assumes them and covers the procedure.)

Two numbers have to be fixed before rounding: the scale $s$, which maps a range of reals onto a range of integers, and the zero point $z$, which decides which integer real zero lands on. Quantizing to $b$ bits,

where $x_{\max}$ and $x_{\min}$ are the largest and smallest values being quantized. Put in words: the spread of the real values, divided by how many integers you can afford, is the width of one step — and the zero point is just how many of those steps sit between the smallest value and real zero. Even at 8 bits there are only 256 integers, so the wider the spread, the coarser the steps.

Quantizing and dequantizing then look like this.

Put in words: divide the value by the step width, round to the nearest whole number, and shove anything that falls off the end back inside the range the bits can hold; to read it back, multiply by the step width again. $q$ is the integer that gets stored and $\hat{x}$ the real that comes back. They do not agree; the gap is at most half a step, $s/2$. The one and only place information is destroyed is that round — precisely the situation in JPEG.

import numpy as np

def quantize(x, bits=8):
    qmax = 2 ** bits - 1
    s = (x.max() - x.min()) / qmax          # the step size
    z = round(-x.min() / s)                 # the grid point real zero lands on
    q = np.clip(np.round(x / s) + z, 0, qmax).astype(np.uint8)
    return q, s, z

def dequantize(q, s, z):
    return s * (q.astype(np.float32) - z)   # the rounded-off fraction is gone

Symmetric, asymmetric, and granularity

Pinning $z$ to zero is symmetric quantization ($s = \max|x| / (2^{b-1}-1)$): cheaper to compute, and a good fit for weights, whose distribution is roughly symmetric about zero. Apply it to something one-sided — activations after a ReLU — and half the integer range goes unused, which is why asymmetric quantization exists.

Granularity matters just as much. Sharing one $s, z$ across a whole tensor (per-tensor) is cheapest, but a single wide-spread row coarsens the step size for everything. So you go finer: per output channel (per-channel), or per block of 128 weights (group-wise). Those scales cost storage themselves — at 4 bits with group size 128 and fp16 scales, an extra 0.25 bits per weight, for an effective 4.25. "4-bit quantization" has never been exactly four bits.

Why weights survive and activations resist

This is the heart of it. Same round, yet weights go down to 4 bits while activations are hard at 8. Three reasons.

First, weights are static. Once training ends the values are fixed, so you can look at all of them and pick $x_{\max}$ and $x_{\min}$ exactly. Activations change with every input, so all you can do is push representative inputs through and collect statistics (calibration) — and if production differs from that sample, the range is wrong.

Second, the shapes differ. Weights are roughly bell-shaped around zero, which suits a uniform grid; activation distributions vary by layer and position.

Third, and decisively, activations have outliers. In large Transformers, particular feature dimensions have been reported to carry activations orders of magnitude larger than the rest, systematically (the LLM.int8() paper, arXiv:2208.07339). Recall that $s$ is set by the maximum. One enormous value coarsens the step size enough that the small values making up the vast majority collapse onto the same grid point. The harm is not the outlier itself but the way it stretches the ruler.

Two ways out: a mixed-precision split computing the outlier dimensions in high precision and the rest in integers, and rescaling that migrates the difficulty onto the weights, where it is easier to absorb (SmoothQuant). Hence the first approach to spread in practice: quantize the weights only and compute in 16 bits — written W4A16.

PTQ and QAT

Methods split on when the quantization happens.

PTQ (post-training quantization) is applied to a finished model. Collect statistics from a small calibration set and round; no training, so even a large model finishes in practical time. Naive rounding tends to break at 4 bits, so smarter rounding was added. GPTQ quantizes column by column and lets the not-yet-quantized weights absorb the error already incurred. AWQ finds the input channels with large activations and pre-scales the weights attached to them. Both refuse to round everything equally.

QAT (quantization-aware training) inserts the rounding into training itself (fake quantization). The forward pass uses rounded values, so the model learns weights that assume low precision. The catch is that the derivative of round is zero almost everywhere, so the backward pass pretends it was not there and passes the gradient through — the straight-through estimator. It costs as much as training, so it is reserved for extreme bit widths and edge-bound small models.

What INT8 and INT4 actually do

The memory arithmetic is straightforward. Seven billion parameters take roughly 14 GB in fp16, 7 GB in INT8, 3.5 GB plus overhead in INT4. Whether it fits on the one GPU you have often turns on that step.

Speed is subtler. With weight-only quantization the integers must be turned back into reals before the matrix multiply, so the arithmetic actually increases. Decode still gets faster because it is bandwidth bound — the saving in bytes read outweighs the extra dequantization.

The flip side follows: push the batch size into the compute-bound regime and that advantage fades. If you cannot say whether you quantized "to make it fit" or "to make it fast", you will misread your own measurements.

Traps in evaluation

Do not look at perplexity alone. Perplexity averages how wrong the next-token prediction is, and an average is blunt. Quantization damage does not spread thinly; it concentrates — following long instructions, generating code, arithmetic with many digits, languages with little training data. One evaluation close to your actual task beats ten general benchmarks.

Keep calibration and evaluation data separate. Calibrate on a distribution and measure on the same one and of course it looks fine. A model calibrated only on English can plausibly miss the activation range on Japanese input.

Change one thing at a time. Turn on weight quantization and KV cache quantization together, see degradation, and you cannot tell which caused it — separate roundings in separate places, and the damage accumulates. Quantization also shifts the output distribution, so prompts tuned previously may stop working the same way.

How this shows up on the job

Quantization is the concern of whoever has to fit a model onto limited GPUs or an edge device and whoever owns the inference bill. The order is settled: try weight-only INT8 or INT4, evaluate on your own task, move to W8A8 or QAT only if that is not enough. Do not open with the most aggressive setting.

The knob names. In llama.cpp, GGUF quantization types (Q4_K_M and friends encode bit width and granularity). With bitsandbytes: load_in_4bit, bnb_4bit_quant_type (nf4 or fp4), bnb_4bit_compute_dtype (get this wrong and things get slower). For GPTQ/AWQ: group_size and the calibration corpus. On the serving side, vLLM's --quantization and --kv-cache-dtype.

Three traps. One: calibrating on a distribution unlike production — if you serve many languages, calibrate on many. Two: trying to keep training an already-quantized model; the low-bit weights are meant to stay frozen, with learning in separately added high-precision parameters. Three: forgetting the effective bit width when sizing the model — 4.25, not 4.

The design-review question is "why doesn't dropping to 4 bits break it?" Two threads. One is that a large model's weights carry redundancy, so no single value's precision governs the output. The other, the real answer, is that the work is done by the cleverness of the rounding — scales split per group, important channels protected, outliers kept from stretching the ruler for everything else. It is not "fewer bits happen to be fine" but "we found ways of removing bits that do not break it". The companion question, "why is INT4 hard for activations?", answers itself in two words: dynamic, and outliers.

Summary

• Quantization rounds continuous values onto grid points; you choose only $s$ and $z$, and the single lossy step is a round (error at most $s/2$) — JPEG's situation exactly
• Weights are static and well behaved, so they compress; activations are dynamic and their outliers stretch the scale
• PTQ rounds afterwards (GPTQ/AWQ round cleverly); QAT bakes the rounding into training
• Weight-only quantization speeds up decode because decode is bandwidth bound — the gain fades at large batches
• Never sign off on perplexity alone; the damage concentrates in particular tasks