Diagnosing Broken Training — Telling Divergence, NaN, and Plateaus Apart

Training breaks in exactly three ways: it diverges, it goes NaN, or it stalls. Built around a symptom-to-cause table, this article shows why divergence is a threshold effect (with the math and an interactive figure), how to pin down where a NaN was born, and how to isolate the cause of a plateau — assuming no prior knowledge.

On the difficulty of training Recurrent Neural Networks

An analogy: diagnosing a car that won't start

Nobody opens up the engine block the moment a car won't start. You check whether the starter turns, then whether there's fuel, then whether there's spark. A mechanic is fast not because their hands are quick, but because they have an order in which to suspect things.

Training a neural network is no different. When the loss curve looks wrong, most people immediately start turning knobs: drop the learning rate by 10×, change the batch size, swap the optimizer. Three hours later the curve looks wrong again. That's the equivalent of replacing the tires because the engine won't start.

The point of this article is to give you that order. Before any theory, get the taxonomy of failures into your head.

Training breaks in exactly three ways

The symptoms look endless, but what's actually happening is always one of these:

1. Divergence: the loss doesn't just fail to fall — it climbs, usually exponentially, reaching absurd values within a few dozen steps.
2. NaN: the loss becomes nan or inf. Once it happens it stays that way forever; there is no recovering.
3. Plateau: the loss doesn't fall, or stops falling partway. Nothing errors out, which makes it the nastiest of the three.

These three have disjoint sets of causes. Divergence is almost always "the update step is too large." NaN is almost always "a value left the representable range." Plateaus have the widest spread — the cause can live in the data, the implementation, or the optimization. So the first move is never to touch a hyperparameter. It's to establish which of the three you are looking at.

The symptom-to-cause table

You narrow things down from the shape of the loss curve plus a couple of logged quantities. "Gradient norm" here means the length of the vector you get by concatenating every parameter's gradient into one long vector. Logging it every step changes how fast you can diagnose anything.

This table isn't something to memorize. It's a device for deciding what to measure first. Notice that every row puts a check before the fix.

Why training diverges (the math)

Gradient descent is one line:

$\theta_t$ is the parameter vector at step $t$ (the weights), $\nabla L$ is the derivative of the loss with respect to each parameter — the slope — and $\eta$ (eta) is the learning rate, the size of one step. In words: move downhill, by a learning-rate-sized amount.

So what happens when the step is too big? Take the simplest possible valley, $L(\theta) = \tfrac{1}{2}a\theta^2$, where $a>0$ controls how steep it is. The slope is $a\theta$, so substituting into equation (1) gives $\theta_{t+1} = (1 - \eta a)\,\theta_t$. Every step just multiplies by the same number, $(1-\eta a)$.

Once the magnitude of that multiplier exceeds 1, the value grows every single step. In words: the instant the learning rate crosses twice the reciprocal of the valley's steepness, that parameter blows up exponentially. And just below the boundary, $(1-\eta a)$ is negative, so the sign flips each step — you bounce from one wall of the valley to the other while creeping downward. That bouncing is exactly the "jittery loss that barely improves" symptom.

What matters is that the threshold depends on $a$, the local steepness — and $a$ differs from layer to layer and parameter to parameter. So a single global learning rate is always too big somewhere and too small somewhere else. Dividing by a running estimate of gradient magnitude, as Adam and RMSProp do, is precisely an attempt to absorb those per-parameter differences in $a$.

The thing to notice in the figure is that divergence is not a gradual degradation. It flips at a threshold. That's why cutting a diverging run's learning rate by 10% accomplishes nothing, and why the standard move is to cut it by a factor of ten.

Warmup — starting the learning rate near zero and ramping it up — works for the same reason. Gradients are large early in training, so the effective $a$ is large; taking smaller steps only in that window keeps you on the safe side of the boundary. That's covered in Learning Rate Schedules — Why Warmup and Cosine.

Where NaNs come from

NaN ("not a number") is the value you get when a result isn't definable as a number. There are essentially only five ways to make one:

• 0 ÷ 0 — a normalization whose denominator collapses: an all-zero mask, a feature with zero standard deviation
• ∞ − ∞ or 0 × ∞ — arithmetic on something that already overflowed to infinity
• log(0) — taking the log of a probability that reached exactly zero; the classic cross-entropy failure
• √(negative) — a variance that went slightly negative from rounding, Adam's $\sqrt{v}$, and friends
• Overflow cascades — a too-large value becomes inf, and arithmetic between infinities produces NaN

NaN has two awkward properties. First, it's contagious: any sum or product involving a NaN is a NaN, so one poisoned parameter contaminates everything downstream on the next step. Second, it isn't equal to itself. x != x is true only for NaN, which doubles as your detection code.

If you're running half precision, the narrow range bites directly. The largest value fp16 can represent is 65504; anything beyond becomes inf. At the other end, gradients smaller than roughly $6\times10^{-8}$ flush to zero. fp32 reaches about $3.4\times10^{38}$, so the same computation survives. When something breaks in fp16 but not fp32, it is almost certainly a range problem. The fix is loss scaling — multiplying the loss by a constant to lift gradients into the representable band — or bf16, a format that makes scaling unnecessary. See Mixed Precision Training — Going Faster in fp16/bf16/fp8 Without Breaking.

Pinning down where the NaN was born (code)

Knowing "we got a NaN" is the easy half. The real work is finding which operation in which layer produced it. Do it in this order.

First, watch cheaply on every step:

loss = model(x, y)
if not torch.isfinite(loss):
    torch.save({"x": x, "y": y, "step": step}, "nan_batch.pt")  # preserve the crime scene
    raise RuntimeError(f"loss={loss.item()} at step {step}")

Be aware that by the time the loss is NaN, the cause is usually one or a few steps earlier. The loss can be perfectly healthy while a gradient goes inf, poisons the parameters, and only then produces a NaN loss. So watch the gradient norm at the same time:

loss.backward()
gn = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
if not torch.isfinite(gn):                 # the clip function returns the total norm
    opt.zero_grad(set_to_none=True)        # throw the poisoned gradients away
    continue                               # skip this batch

clip_grad_norm_ returns the global gradient norm, so clipping and monitoring come in the same call. Simply skipping batches whose norm isn't finite prevents a large share of the "one outlier destroyed a six-hour run" accidents.

When that still doesn't localize it, there's a last resort:

torch.autograd.set_detect_anomaly(True)   # several times slower; only while hunting

With this on, the backward op that produced the NaN raises an exception, with a stack trace back to the forward line that created it. It's dramatically slower, so never leave it on.

The three faces of a plateau

Plateaus have the widest spread of causes, so split them further.

(a) It never moves at all. This is not an optimization problem — it's almost always an implementation bug. Gradients aren't reaching the parameters; some parameters were never handed to the optimizer; something is still frozen with requires_grad=False; zero_grad() is in the wrong place. The fastest way to tell is the single-batch overfit test in the next section.

(b) It flattens at a particular value. If that value is close to $\ln(\text{num classes})$ — about 2.30 for ten classes — the model has learned nothing beyond "predict every class equally." Suspect broken input-label correspondence first. Shuffling images and labels independently in the data loader, or a collate implementation that misaligns the ordering, throws no error and produces exactly this symptom.

Saturated activations look the same. If every input to a ReLU is negative, its gradient is zero and the weights behind it never update again (a dead ReLU). Sigmoid and tanh are similarly flat for large-magnitude inputs. Weight initialization and normalization layers exist precisely to avoid this — see Weight Initialization and Regularization.

(c) Train loss falls while validation loss climbs. Nothing is broken; this is overfitting, the model entering the phase where it memorizes the training set. The fix isn't to repair training but to rebalance capacity, regularization, and data.

The isolation procedure (run it top-down)

Diagnosis runs cheapest and most decisive first.

1. Check the loss at initialization. A randomly initialized classifier should predict all classes equally, so cross-entropy should start at $\ln C$ where $C$ is the number of classes. If it's far off, your output layer or your loss function call is wrong. This costs zero training steps — the cheapest test there is.
2. Overfit a single batch. Train on the same few dozen samples for hundreds of steps and see whether the loss approaches zero. If it doesn't, the problem is neither data nor learning rate — it's an implementation bug. If it does, gradients flow correctly and you can restrict all further suspicion to data and optimization. This takes minutes.
3. Turn off all regularization. Temporarily disable augmentation, dropout, and weight decay, then see whether the symptom reproduces on the simplified run. If it doesn't, your culprit is in that set.
4. Force fp32. If that fixes it, it's a range problem; if not, it's an algorithmic one. One run, clean split.
5. Fix the seeds and reproduce. A bug you can't reproduce is a bug you can't fix. Pin the data order, the initialization, and the dropout seeds, and confirm the same symptom at the same step before changing anything.

The property that makes this order worth following is that each step roughly halves the remaining candidates. Guessing at hyperparameters has no such property.

How this plays out in practice

Who hits this, and when. Anyone who trains models — ML engineers, researchers, people running foundation-model pretraining — meets all three failures the first time they run a new dataset or configuration, and again every time they scale up. The moment right after making a model bigger deserves special caution: a learning rate that was stable on the smaller model frequently diverges unchanged.

Parameters and tools you'll actually touch. The learning rate lr; the number of warmup steps; the gradient clipping threshold (max_norm in torch.nn.utils.clip_grad_norm_ — values around 1.0 are widely used for language models); Adam's eps (in fp16 the default 1e-8 can be small enough that the denominator collapses, so some teams raise it to around 1e-6); AMP's GradScaler. For monitoring, TensorBoard or Weights & Biases — and always put gradient norm and learning rate on the same screen as the loss. With those three curves in front of you, the table above becomes something you can read off directly.

Pitfalls that turn into incidents.

• Where you clip. Under AMP, clipping gradients that are still scaled makes the threshold effectively meaningless. Call scaler.unscale_(optimizer) first.
• Giving up on reproducing a spike. "It only happens sometimes" guarantees it will happen during the long production run. Build in the machinery to dump the batch and seed at failure time from day one.
• Overwriting checkpoints. Saving over a good checkpoint with post-NaN weights leaves you nowhere to roll back to. Check torch.isfinite before saving, and keep generations.
• Changing several things per experiment. Even when it works, you won't know what worked — and the next occurrence starts from scratch.

What gets asked. "Your loss suddenly went NaN. What do you check?" is a staple in both real work and interviews. The answer that lands isn't a list of remedies but an order: (1) fix the reproduction conditions, (2) look at loss and gradient norm together to identify the exact step, (3) rerun in fp32 to separate range problems from algorithmic ones, (4) save the offending batch and build a minimal repro. Having the procedure counts for more than knowing the fixes.

Summary

• Training breaks in three ways — divergence, NaN, plateau. Establish which one before doing anything else.
• Divergence is a threshold effect in the learning rate. Past the boundary it grows exponentially, so the fix is an order of magnitude, not a nudge.
• There are only five ways to make a NaN. Logging the gradient norm every step and dropping non-finite batches prevents most of them.
• Plateaus have the broadest causes. The single-batch overfit test separates implementation bugs from everything else first.
• Diagnose cheapest-and-most-decisive first, with a procedure where every step halves the candidate set.

Most of the time, a loss that won't fall isn't a modeling problem — it's a measurement problem. Log loss, gradient norm, and learning rate from the very first run, and the table above works as written.