Learning Rate Schedules — Why Warmup and Why Cosine

The learning rate is not a fixed number — it is a curve you design across the whole run. Why we deliberately start slow (warmup), why we come down along a cosine, and what else has to move when batch size changes. Equations, live figures, PyTorch code, and the mistakes that actually break runs.

Attention Is All You Need

An analogy: pulling out of an icy lot and backing into a space

You are backing a car into a space across a snow-covered parking lot. Right after you pull out you have no idea whether the surface is ice, so you ease onto the throttle and feel for grip. Across the open stretch you carry real speed. As the lines come up you slow down — carry that speed in and you overshoot, correct, overshoot again, and never settle.

Training a neural network moves through those same three phases. Easing onto the throttle is warmup, the open stretch is the peak learning rate, and slowing for the fine adjustment is decay (cosine and friends). A learning rate schedule is that throttle work, written down as a function of the training step.

What the learning rate actually decides

Training is one thing repeated: nudge the weights. A single nudge looks like this.

Here $\theta_t$ are the parameters at step $t$, $\nabla L(\theta_t)$ is the gradient — the arrow giving the direction the loss $L$ decreases and how steeply — and $\eta_t$ is the learning rate. Equation (1) says only this: from where you stand, take one step downhill of size $\eta_t$.

Written out in words: new weights = current weights − (stride) × (downhill direction). The minus sign is what makes this a descent rather than a climb, and the multiplication is a division of labour — the gradient chooses the direction, the learning rate chooses how far you go in it. The gradient itself is unpacked in Loss Functions and Optimization.

The crucial part is that the gradient tells you only a direction and a steepness — nothing about how far you must travel to reach the bottom. That distance is what the learning rate supplies, and the right value changes with where you are. Hence the subscript $t$ on $\eta$: not a constant, but a value chosen fresh each step.

What goes wrong with one fixed stride

Too small, and you head the right way but never arrive. GPU hours are finite, so that is failure in practice.

Too large, and you clear the valley in one stride and land higher up the opposite slope. The gradient there still points into the valley, so you leap across again. Each crossing widens the swing until the loss diverges into NaN.

The awkward part is that you cannot see the boundary in advance. Feel for it in the figure below: raise the slider and convergence speeds up — until one notch past a threshold the ball flies out of the valley entirely.

Two things fall out. There is a ceiling on a good learning rate, and the closer you get to the bottom, the smaller the right stride becomes. So: big early, small late. That is where decay comes from.

Why not simply run a small learning rate throughout and stay safe? Training does still work that way, but the loss you reach on the same compute budget is clearly worse. A schedule is not a device for avoiding crashes; it is a device for reaching the lowest loss within a fixed number of GPU hours.

And there is one more twist. Almost no modern run uses a plain downward slope — it deliberately starts small, climbs, and only then comes down. That odd hump is warmup plus cosine.

warmup — deliberately crawling for the first few hundred steps

Warmup is simple. Start near zero and raise the rate linearly to the peak $\eta_{\max}$ over a chosen number of steps $T_{\text{warm}}$.

Equation (2) says the rate rises in proportion to how far through the warmup you are. With a 1,000-step warmup, step 100 sits at 10% of the peak. That is the whole rule.

Put in words, $t / T_{\text{warm}}$ is a progress meter that runs from 0 to 1 across the run-up, and the learning rate at any moment is simply that meter's reading times the peak value.

The real question is why anyone would do this. Three reasons, and they stack rather than compete.

1. The starting weights are random noise. The loss is high and the gradients are large, but nothing guarantees they point somewhere useful. Take a big step from there and you can land worse off than random initialization left you. Treat the first few steps as gathering information rather than making progress.

2. Adam's internal statistics are still estimated from a handful of gradients. Adam tracks a running estimate of the squared gradient and divides by its square root to set a per-parameter effective stride. Right after the start that estimate is small and noisy, and dividing by a small number sends the effective learning rate spiking. The RAdam paper (Liu et al., 2020) singled out this early variance as the main reason warmup helps. No single explanation has won out, but the phenomenon — Adam-family optimizers tend to break early without warmup — reproduces widely.

3. The bigger the batch, the more the first step counts. A larger batch means less gradient noise, so every parameter moves the same way at once. Marching everything together in a direction not yet shown to be correct is exactly the risky case.