How Numbers Are Represented — From FP32 to FP8 and INT4

Open up the sign, exponent and mantissa fields and one rule falls out: the exponent buys range, the mantissa buys precision. From there — why bfloat16 was invented, why FP8 ships in two flavours, and what integer quantization actually destroys. Ends with which format to pick for training versus inference, and how to notice degradation before your users do.

Putting a finite number of ticks on the real line

A computer has a finite number of bits. "Representing a real number" therefore means placing finitely many ticks on the real line and rounding to one of them. The only freedom in the design is where to place the ticks.

There are essentially two placements. Evenly spaced ticks give you fixed-point and integer formats. Ticks that are dense near the origin and sparse far from it give you floating point — a deliberate trade in which absolute error is sacrificed to hold relative error constant. That trade is what lets scientific computing handle quantities spanning dozens of orders of magnitude.

Opening up a floating-point number

A floating-point format divides its bits into three fields: a sign $s$ of one bit, an exponent $E$ of $e$ bits, and a mantissa $f$ of $m$ bits. The value is

$s$ is the sign (0 for positive), $E$ is the unsigned integer stored in the exponent field, $b$ is the bias $2^{e-1}-1$ that lets exponents go negative, and $f$ is the mantissa field read as a fraction, $0 \le f < 1$. The leading 1 is never stored: for a normalized number it is always 1, so it is implicit, and the mantissa effectively gains a bit for free.

The same statement in words: the sign picks the direction, the power of two fixes roughly how big the number is, and the fraction $1+f$ says where between that power of two and the next one you actually land. It is scientific notation — $1.23 \times 10^{4}$ — rewritten in binary, with the exponent field holding the "$\times 10^{4}$" part and the mantissa field holding the "$1.23$" part.

Two properties follow immediately from that form.

Range is set by the exponent width. The largest representable value is roughly $2^{2^{e-1}}$, so each extra exponent bit squares the maximum — it doubles the number of decades you can express.

Precision is set by the mantissa width. The relative gap between neighbouring representable numbers is $2^{-m}$, known as the machine epsilon. At $m = 23$ that is about $1.2 \times 10^{-7}$, roughly seven significant digits.

Put in words, the gap to the next representable number is not a fixed absolute amount but always about the same fraction of the number itself, roughly one part in $2^m$. So one more mantissa bit halves the relative error, and it does so just as much near $10^{-30}$ as near $10^{30}$.

Exponent buys range; mantissa buys precision. Those two lines are the skeleton of this article — everything below is derived from them.

FP32, FP16, and the accident that happens during training

The common formats, side by side:

Look at FP16. With only five exponent bits it stops at 65504 on the way up, and at around $6 \times 10^{-5}$ for normalized numbers on the way down. Gradients during training routinely go below that. A gradient of $10^{-8}$ becomes exactly zero in FP16 — underflow. A zeroed gradient kills everything downstream of it in the backward pass, and training quietly stops improving.

The workaround is loss scaling: multiply the loss by a large constant (say $2^{15}$) before the backward pass, then divide the gradients by the same constant before the update. It shifts the whole gradient distribution into the representable band. It works, but too large a scale overflows and too small a one underflows, so the scale has to be tuned automatically at runtime. FP16's awkwardness comes from range, not from precision.

Why bfloat16 was invented

The idea that followed was to re-divide the same sixteen bits. bfloat16 gives the exponent 8 bits — exactly what FP32 has — and pays for it by cutting the mantissa to 7.

The consequences are clean. First, the representable range is identical to FP32, so nothing overflows or underflows on conversion and loss scaling becomes unnecessary. Second, converting to and from FP32 is just truncating the low 16 bits of the mantissa, which is nearly free in both hardware and software.

The price is precision: about 0.8% relative error, an order of magnitude coarser than FP16's 0.1%. So why does training still converge? This is the heart of the design.

• Gradients, and quantities like Adam's second moment, differ in magnitude by many orders across layers and parameters. Insufficient range makes values disappear entirely — unrecoverable.
• A weight update, by contrast, is an average over many samples in a minibatch, and rounding errors point in random directions, so they largely cancel statistically rather than accumulating into a fatal bias.

Training, in other words, is fragile to a lack of range and robust to a lack of precision. bfloat16 is a format whose bit budget matches that asymmetry.

Why FP8 comes in two flavours

Cut down to eight bits and the tug-of-war between exponent and mantissa becomes impossible to ignore, so FP8 shipped as two types from the start.

• E4M3 (4 exponent, 3 mantissa): narrower range, relatively better precision. Used for forward-pass quantities like activations and weights, whose magnitudes are comparatively uniform.
• E5M2 (5 exponent, 2 mantissa): coarse precision, wide range. Used for gradients, which span many orders of magnitude.

The same eight bits, split differently depending on what you are representing. It is the "exponent buys range, mantissa buys precision" rule turned directly into a product decision.

Integer formats and quantization

INT8 and INT4 use evenly spaced ticks. Getting back to a real number takes a scale $s$ and a zero point $z$:

$q$ is the stored integer, $s$ is the width of one tick, and $z$ is the integer that maps to real zero. Fixing $z = 0$ is symmetric quantization; letting it move is asymmetric. One $s$ per tensor is per-tensor quantization, one per output channel is per-channel, one per group of a few dozen elements is group-wise — each step reduces error and adds metadata.

That formula in words: read the stored integer as a tick number on a ruler, subtract the tick that sits at real zero, then multiply by the real width of one tick. Counting ticks and turning the count into a length — nothing more. Everything the format knows about the original numbers lives in those two constants, $s$ and $z$.

The strength of integers is that they spend nothing on an exponent. When values cluster in a narrow band, uniform ticks resolve them more finely than floating point at equal width, and neural network weights do cluster around zero. The weakness has the same origin: a single outlier drags $s$ upward and collapses the vast majority of values onto a handful of ticks. That structure is exactly why outlier channels in activations are the notorious hard case in quantization.

What quantization destroys

(The practical side — choosing scales and zero points, PTQ vs QAT, handling outliers — is covered in LLM quantization from scratch. This article stays on the formats themselves.)

Two kinds of error, and they trade against each other.

Resolution error. With a uniform step $\Delta$, the error is roughly uniform on $[-\Delta/2, \Delta/2]$ with variance $\Delta^2/12$. Add one bit and $\Delta$ halves, error power drops fourfold, and SNR improves by about 6 dB — the classic "one bit equals 6 dB" rule from signal processing.

Clipping error. Values outside the representable range get pinned to the endpoints. Widen the range and resolution coarsens; narrow it and clipping grows. Calibration is nothing more than choosing, from real data, the range that minimizes the sum of these two errors.

Seen information-theoretically, quantization discards information irreversibly (information theory). As in image compression, discarding it skilfully leaves perception — or prediction — almost untouched. Feeling how JPEG behaves as the step coarsens is a good way to internalize the INT4 argument.

How this is used in practice

Which format for training. The default is mixed precision with BF16: matrix products run in BF16 while the master weights are kept in FP32. The reason is concrete — when an update is small relative to the weight, BF16's 7-bit mantissa rounds the addition away and the update vanishes entirely. Operations prone to cancellation in long sums — normalization statistics, softmax, the loss itself, gradient reductions — are also kept in FP32. In PyTorch the entry point is torch.autocast(device_type='cuda', dtype=torch.bfloat16); only if you choose FP16 do you additionally need loss scaling via torch.amp.GradScaler.

Which format for inference. The deciding factor is the bottleneck resource from the memory wall. Token-by-token decode is bandwidth bound, so quantizing only the weights to INT4/INT8 while activations stay BF16 (weight-only quantization) cuts bytes moved and pays off directly. Prefill processes the prompt in a batch and is compute bound, so schemes like W8A8 that make the arithmetic itself cheaper are what help there. The same model wants different formats in different phases.

Metrics that reveal degradation. Task accuracy alone will not tell you the model broke. In practice, combine:

• KL divergence against the original model's output distribution. Cheap to compute on a handful of prompts and highly sensitive
• Perplexity delta — look at the change from the unquantized model, not the absolute number
• Per-layer cosine similarity of activations, which localizes exactly where things fell apart
• Activation maxima and a list of outlier channels; saturating channels are usually the culprit
• Counts of inf and NaN, plus gradient-norm histograms during training

Pitfalls that turn into incidents

• Quantized and it didn't get faster. Usually either integers are being converted back to float around every operation (dequantization is the bottleneck) or no kernel exists for that format and a fallback was selected. Profile before believing anything.
• Calibration data that doesn't match production. Choose the range on the wrong inputs and production clips. Include the edges: long inputs, other languages, unusual symbols.
• Double quantization. Quantizing an already-quantized model accumulates error. Go back to the original weights when you can.
• KV cache quantization is a separate problem. It has worse outliers than weights and degrades faster at long context. Do not reuse the weight settings blindly.

The interview version: "What is the difference between BF16 and FP16?" — the bit split (8 vs 5 exponent, 7 vs 10 mantissa), the resulting range/precision trade, and the operational consequence that FP16 needs loss scaling and BF16 does not. If the follow-up is "so which do you use for inference?", answer in terms of the bottleneck resource and which kernels actually exist, and the conversation lands.

Summary

• Floating point is $x = (-1)^s 2^{E-b}(1+f)$: the exponent sets range, the mantissa sets precision
• FP16's weakness is range, not precision; gradient underflow is what made loss scaling necessary
• bfloat16 matches FP32's exponent to keep range and pays with mantissa bits — training is fragile to range, robust to precision
• FP8 exists as E4M3 and E5M2 because forward activations and gradients need different ranges
• Integer quantization is uniform; its error is resolution plus clipping, and minimizing that sum is calibration