Numerical Pitfalls — Cancellation, Rounding, and logsumexp

Where "the loss went nan three hours into the run" actually comes from, built up from nothing: how rounding enters, how the condition number amplifies it, and why subtracting two close numbers is so destructive. It all converges on logsumexp — the one trick sitting inside every softmax and cross-entropy implementation.

Why 0.1 + 0.2 isn't 0.3

Evaluate 0.1 + 0.2 in any language you like. You get 0.30000000000000004. That is not a bug, it is the specification. A tenth divides cleanly in base 10 but repeats forever in base 2, so it has to be rounded to fit a finite number of bits. That's all that happened.

Most days you can ignore that $4\times10^{-17}$. The trouble is that an error this small grows by orders of magnitude when it passes through certain operations — and when it does, it usually only becomes visible as "the loss went nan three hours into the training run." The cause sits far away from the symptom, so guessing at fixes burns days.

This article walks through where the error is born, where it gets amplified, and how to write things so it stays small.

An analogy: a tape measure that gets coarser to the right

Think of a floating-point number as a tape measure with a fixed number of significant digits. Near 1 it resolves down to 0.0000001; near ten million it only resolves down to 1. The total number of tick marks is fixed, so buying reach costs you resolution.

The exponent decides which stretch of the tape you're standing on; the mantissa decides how fine the ticks are there. The bit-level breakdown per format is opened up in how numbers are represented, from FP32 to FP8 and INT4; the one consequence to carry forward is this: absolute error varies enormously with magnitude, but relative error is roughly constant everywhere. Every pitfall below follows from that single line.

Intuition: error enters multiplicatively, not additively

The distance from 1 to the next representable number is called the machine epsilon — the headline figure for how fine that format's ticks are.

Each arithmetic operation returns the true answer rounded to the nearest tick, so a single operation costs you at most half a machine epsilon in relative terms. Nobody is hurt by one operation. You get hurt when the cost accumulates over many of them, or when it passes through the amplifier in the next section.

The same story bites integers. Every consecutive integer is exactly representable only up to 16,777,216 (two to the 24th) in float32, and only up to 2,048 in float16. Past that the ticks are spaced wider than 1, so adding one stops changing the value. Count rows or cumulative tokens in an fp16 tensor and the counter freezes at 2048. Nothing raises an exception.

The amplifier: condition number

Given the same input error, some problems move a lot and others barely budge. That multiplier is the condition number. For solving $Ax=b$ it is $\kappa(A)=\lVert A\rVert\,\lVert A^{-1}\rVert$; for a symmetric matrix it reduces to the ratio of eigenvalue magnitudes, $|\lambda_{\max}|/|\lambda_{\min}|$.

There is one rule of thumb to use it: relative error in, times the condition number, is roughly relative error out. A condition number of $10^k$ costs you about $k$ significant digits. float32 only holds about seven, so solving a problem with condition number $10^6$ in float32 leaves you exactly one digit of answer. The formula can be perfectly correct and still have nothing left.

The everyday version is finding where two nearly parallel lines cross: tilt one of them a hair and the intersection flies off somewhere else entirely. The same thing happens in the normal equations when two features are strongly correlated — and it is why ridge regression adds a small value along the diagonal. That is a regularizer, but it is equally an operation that lowers the condition number.

Geometrically: a matrix turns the unit circle into an ellipse, and the condition number is the ratio of its long axis to its short axis. The flatter the ellipse, the more a tiny wobble along the short axis gets stretched when you go back the other way — that is, when you apply the inverse.