Chapters 1 to 3 settled which way to move. Two things are left: how far to step each time (the learning rate) and when to stop.

They look like separate questions. Chapter 1 deferred the first and Chapter 2 the second, both to "Chapter 4". Arriving here, the two turn out to be two grips on the same knob — that is this chapter.

1. The learning rate has a ceiling

The update is

Watch what happens on the one-dimensional quadratic $L(w) = \tfrac{1}{2}\lambda w^2$. With $\nabla L = \lambda w$, one update is

$$w \;\leftarrow\; w - \eta \lambda w \;=\; (1 - \eta\lambda)\, w$$

It is simply multiplied by $(1-\eta\lambda)$ every time, so after $t$ steps it is $(1-\eta\lambda)^t w_0$. That heads for zero exactly when $|1-\eta\lambda| < 1$, i.e.

In several dimensions the sharpest direction decides (the largest eigenvalue $L_{\max}$ of the Hessian). For nothing to diverge, $\eta < 2/L_{\max}$. Cross it and that one direction is amplified until everything breaks.

Measured on $f(x,y) = (x^2 + 10y^2)/2$ from Chapter 0-2. The curvatures are $1$ and $10$, so the ceiling is $2/10 = 0.2$. Starting from $(10,1)$, steps to get within radius 0.01 of the origin:

The best value sits well below the ceiling (0.18), and performance collapses as the ceiling nears: 459 steps at $0.199$, 4,603 at $0.1999$. It becomes useless long before it falls off the cliff.

2. Why one step size cannot suit every direction

The same $\eta$ multiplies every direction, but curvature differs by direction. A direction with curvature $\lambda$ is multiplied by $(1-\eta\lambda)$ each step.

• Raise $\eta$ and the sharpest direction starts oscillating at $|1-\eta\lambda_{\max}|$
• Lower $\eta$ and the flattest direction barely moves, at $1-\eta\lambda_{\min} \approx 1$

The best is where those balance. Solving $1-\eta\lambda_{\min} = -(1-\eta\lambda_{\max})$:

For the example, $\eta^{*} = 2/11 = 0.1818$. Sweeping, the minimum came out at $\eta \in [0.180,\ 0.186]$ with 35 steps. Estimating the count from the contraction $9/11 = 0.818$ gives $\lceil \log(0.01/10.05)/\log(0.818) \rceil = 35$ — matching the measurement.

$\kappa$ is the condition number. The larger it is, the closer the contraction sits to 1 and the less progress any number of steps makes. What makes learning rates hard is not model size but $\kappa$.

3. Momentum — cancelling the back-and-forth

The zigzag is the component that reverses every step. Add the steps up and it cancels.

$\beta$ says how much of the previous direction to carry. Reversing components cancel; components that keep pointing the same way (down the valley) accumulate.

Measured on the same problem with $\eta = 0.05$ fixed:

$\beta = 0.5$ turns 135 into 59. But past 0.9 it gets slower: carry too much and it cannot turn where it needs to, so it overshoots. As with the learning rate, more is not better.

4. Adam — a different step size per direction

The condition-number problem came from using one $\eta$ everywhere, so vary it: divide each coordinate by a running average of its squared gradient.

($\hat m, \hat v$ correct the bias from starting at zero; the details are in the article.) Coordinates that fire often get a large $\sqrt{\hat v}$ and a shortened step; rarely-moving coordinates get a longer one.

It promises no speed, though. On the same problem:

Plain gradient descent wins here, because the curvature is known and $\eta^{*}$ can be chosen. Adam's worth is not speed but landing somewhere reasonable without tuning, on problems whose curvature you do not know. In deep models, where gradient magnitudes differ by orders of magnitude between layers, "reasonable without tuning" is what matters.

5. When to stop — the step count is capacity too

Chapter 2 listed early stopping among the remedies for overfitting, on the grounds that "the number of steps is a form of capacity." Here is the measurement.

Same data as Chapter 2 ($\sin(2\pi x) + \varepsilon$, $\sigma = 0.25$, 20 points, degree 15), running gradient descent with no regularisation at all:

Training error falls to the end; test error bottoms out at $t = 1{,}370$ and climbs. The U-shape Chapter 2 produced by raising the degree appears here by raising the step count.

The right-hand column says why. $\lVert w \rVert$ grows the longer you run. Limiting the steps was limiting $\lVert w \rVert$.

Applying L2 (Chapter 2, equation (3)) to the same data and sweeping $\lambda$:

The floors differ by 0.0072 and the $\lVert w \rVert$ values nearly agree. The knobs — a step count and a $\lambda$ — have nothing in common, and they are doing the same thing.

A common misreading: "as long as it doesn't diverge, a bigger learning rate is faster"

Look again at section 1. The ceiling is $0.2$.

The cliff is at $0.2$, but performance starts falling from $0.18$. $0.1999$ does not diverge — it converges perfectly well — and takes 131 times as long as the best.

Equation (3) explains it. Raising $\eta$ drives the sharpest direction's factor $1 - \eta\lambda_{\max}$ toward $-1$ (oscillation), and as $\eta \to 2/\lambda_{\max}$ the contraction approaches 1: it stops shrinking. The best is where oscillation and sluggishness balance, at $\eta^{*} = 2/(\lambda_{\min}+\lambda_{\max})$ — not at the ceiling.

Momentum has the same shape (section 3): best at $\beta = 0.5$, and 678 steps at $0.99$. "As large as possible without diverging" is wrong for both knobs.

Exercises

The answers are folded away. Work them first.

Exercise 1. A problem with curvatures $\lambda_{\min} = 0.5$ and $\lambda_{\max} = 50$. (a) The ceiling on $\eta$. (b) The best $\eta$. (c) The resulting contraction, and the steps needed to cut the error to $1/100$.

Exercise 2. You halve the learning rate. (a) In section 5's table, where does the test-error minimum move? (b) What happens to its height? (c) What follows from that?

Exercise 3. The loss keeps falling while the validation error starts rising. (a) What is happening? (b) Will lowering the learning rate fix it? (c) Name three things to do.

What this chapter does not say

• Stochastic (minibatch) gradients are not included. Every measurement here uses the full-data gradient. The minibatch noise measured in Chapter 0-3 ($\sigma/\sqrt{B}$) is often said to act as a regulariser in itself; that is not measured here.
• Learning-rate schedules (decay, warmup, one-cycle) appear only as "machinery for chasing a moving ceiling"; which to use when is not covered.
• Adam's bias correction, the role of $\beta_2$, and its interaction with weight decay (AdamW) are left to the article.
• "Early stopping = L2" is not an exact identity. It is a classical correspondence under a quadratic approximation, and what the text shows is that one measured setting comes out close. Nothing here claims it holds unchanged for a nonlinear model.
• Every number was computed and checked (the learning-rate sweep against the theoretical $\eta^{*}$, the 35 steps predicted from the contraction against the 35 measured, the momentum and Adam sweeps, the early-stopping trace and the L2 sweep, and the comparison of the two optima).

Going deeper

• Gradient descent from scratch — the geometry of learning
• Adam from scratch — two moving averages and the bias correction nobody explains
• Beyond SGD — Adam, second-order methods, constrained optimisation
• Convexity and optimisation — why deep learning works although it is not convex