Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking

Halve the bits and training gets faster — right up until your gradients quietly turn into zeros or infs. We start from the two ways it breaks, work out what loss scaling actually does, why bf16 made it unnecessary, and what came back with fp8. Ends with the PyTorch AMP knobs and the clipping-order mistake that fails silently.

Mixed Precision Training

What happens when your ruler gets coarser

Say you have two tape measures. One is marked in millimetres and reaches three metres. The other is marked in centimetres and reaches thirty. Both have roughly the same number of marks on them. Measuring finely and measuring far are a trade, and once the total number of marks is fixed, buying one means selling the other.

Numbers in a computer work the same way. Drop from 32 bits to 16 and memory halves while matrix multiplies land on dedicated hardware (Tensor Cores) and speed up. In exchange, either the marks get coarser, or the reach gets shorter, or both.

The trouble is that the numbers flowing through training span an enormous range. Loss is around 1, weights around 0.01, gradients around 0.0001 — sometimes 0.00000001. The moment a value falls off the end of your ruler it silently becomes 0 or inf. Mixed precision training is the collective name for the tricks that let you use the coarse ruler without that accident.

What exactly is being "mixed"

The name is mixed because you do not put the whole model in low precision. Numbers during training play at least three different roles, and each needs a different amount of precision.

1. Computing (matmuls, convolutions) — inputs can be low precision. Essentially all of the speedup lives here
2. Accumulating (sums, reductions) — thousands of terms get added, so dropped digits compound
3. Storing (the weights themselves) — nudged slightly over tens of thousands of steps, so small differences must not be thrown away

Mixed precision assigns these to different types: compute in fp16 or bf16, accumulate and store in fp32. GPU matmul units are already built to multiply in low precision and add in fp32, so 1 and 2 happen for free. The one you have to be deliberate about is the third — the so-called master weights.

The 2017 paper Mixed Precision Training proposed exactly this trio (fp32 master weights, loss scaling, fp32 accumulation) and showed it matched fp32 accuracy across image classification, translation and speech. What today's frameworks give you by default is largely that paper, productised.

The two ways training breaks

fp16 covers roughly 0.00006 up to 65504 (symmetric on the negative side). Values that leave that window break in two different ways.

Falling off the top (overflow) gives you inf. An inf soon meets something like inf - inf and becomes nan, and nan survives every multiply and add, so one of them infects the whole model within a few steps. That is the run where loss jumps to nan and never comes back. It breaks loudly, which at least means you notice.

Falling off the bottom (underflow) is nastier: the value quietly becomes 0. No error, no warning. A weight whose gradient is zero is simply not updated, and the only symptom is "accuracy is somehow not improving". The gradient histogram in the original paper showed a substantial share of gradient magnitudes sitting below what fp16 can represent at all. Left alone, they vanish — that is the starting point.

What makes this hard is that it looks exactly like a learning-rate problem. Too small a step and you crawl; too large and you fly off. Get that behaviour into your hands with the figure below. What mixed precision adds is a step that becomes 0 or inf because of the number format, with the learning rate untouched.

Exponent and mantissa — fp16 and bf16 are two answers to the same 16 bits

Why values become 0 or inf is obvious once you look at how the bits are split. A floating-point number has three fields.

Here $s$ is the sign (0 positive, 1 negative), $e$ is the integer stored in the exponent field, $b$ is a per-format offset called the bias, and $f$ is the fraction in $[0,1)$ held by the mantissa. In words: a power of two sets the order of magnitude, and the "1.xxx" part sets how finely you land within it. Exponent bits buy reach; mantissa bits buy resolution.

Given the same 16 bits, fp16 and bf16 cut in opposite directions. fp16 shrinks the exponent from 8 to 5, giving up reach to keep 10 mantissa bits of resolution. bf16 keeps all 8 exponent bits — matching fp32's reach exactly — and pays for it by cutting the mantissa to 7. For a closer reading of the bit layouts, see Number Formats — from FP32 to FP8 and INT4.