Distillation vs. Quantization vs. Pruning — Three Roads to a Smaller Model

There are three roads to a smaller model: coarsen the number grid (quantization), remove weights outright (pruning), or rebuild the thing at a smaller size (distillation). A head-to-head comparison on compression ratio, accuracy, and implementation cost — and why, when you stack them, training-based methods go first and quantization goes last.

Distilling the Knowledge in a Neural Network

Three ways to lighten a load

Say you want to move house with fewer boxes. You have three options.

You can lower the fidelity of what you keep: re-save every photo in the album at a coarser resolution. The count stays the same; only the bytes shrink. You can throw away what you never use: the pot you haven't touched in three years goes, and the pile physically shrinks. Or you can look at everything and rebuild: lay it all out, decide what this life actually requires, and assemble a smaller kit from scratch.

Shrinking a neural network maps onto those three options with surprising precision. In order: quantization, pruning, and distillation. This article puts all three on the same table. The goal isn't to go deep on any one of them — it's to answer two practical questions: which one fits your situation, and if you use several, in what order.

What each one actually removes

Let's fix the vocabulary first, because the three techniques delete genuinely different things.

Quantization lowers the bit width of the numbers that represent weights and activations. A weight stored in 16 bits (fp16) gets stored in 4 instead. The parameter count does not change at all, and neither does the shape of the model. Only bytes-per-weight goes down.

Pruning removes weights — either by zeroing individual entries, or by deleting whole heads, channels, or layers. The surviving weights keep their full precision; there are simply fewer of them.

Distillation trains a different, smaller model (the student) using the outputs of the large one (the teacher) as its target. Layer count, hidden width, vocabulary, even the architecture itself are all yours to choose. You aren't modifying the original model; you're rebuilding it.

In one line: quantization cuts precision, pruning cuts count, distillation rebuilds. That difference is what produces every entry in the table below.

Head to head on three axes

The row to stare at is the second-to-last one. "Smaller" and "faster" are not synonyms. Quantization shrinks what sits in memory and shrinks the bytes you read per step, so in the bandwidth-bound generation phase it translates directly into speed. Pruning, as we'll see, buys you nothing at all unless the zeros you created are in a form where the hardware can actually skip the work. Get this wrong in a design doc and you'll discover it only after the implementation lands and nothing improved.

Before going further, it's worth feeling what "coarsening the grid" does. The mechanism is essentially the same one JPEG uses.

Note the shape: it doesn't degrade smoothly. It holds up to a point, then drops off a cliff. All three techniques behave this way.

Quantization — the cheapest option, and the one to try first

Underneath, quantization is a division and a rounding step that maps a range of real numbers onto an evenly spaced grid. For the simplest symmetric scheme:

Here $w_i$ is the original weight, $b$ is the bit budget, and $s$ is the grid spacing (scale). In words, equation (1) says only this: pick the spacing so that the largest weight in the group lands exactly on the largest representable integer, then round everything else onto that grid. What you store is the rounded integers plus one $s$ per group.