Chapter 1 showed that the direction to move a weight is set by $p - y$, the gap between prediction and truth. That was with one layer. Stack layers and only the top one knows the gap: the weights below never touch the output, so they have no way of knowing how wrong they were.

How does the gap get down there? Chapter 0-2 already answered it. Multiply along a chain, add across a branch — the chain rule, and nothing else. This chapter writes that rule in the shape of layers, checks it against numerical differentiation, and then measures the case where it fails to arrive (vanishing gradients). No new principle appears.

1. The forward pass

Write layers $l = 1, \dots, L$ as

with $a^{(0)} = x$ the input and $a^{(L)}$ the output. $\varphi$ is applied componentwise (Chapter 0-2, case (b)).

The loss $\ell$ is a function of the output alone, and there are $L$ sets of weights. What we need is the gradient with respect to every $W^{(l)}$.

2. The chain rule in the shape of layers

Define one key quantity: how much the loss moves when that layer's input $z^{(l)}$ moves.

$$\delta^{(l)} \;=\; \frac{\partial \ell}{\partial z^{(l)}} \;\in\; \mathbb{R}^{n_l}$$

At the output layer, the form from Chapter 1 carries over directly (for cross-entropy with softmax):

Now descend. The only route by which $z^{(l)}$ reaches the loss is through $a^{(l)} = \varphi(z^{(l)})$ into $z^{(l+1)}$, so applying equation (7) of Chapter 0-2 ($J_{f\circ g} = J_f J_g$):

That single line is all of backpropagation. Read it in two halves.

• Multiplying by $W^{(l+1)\top}$ redistributes the layer above's error in proportion to the weights. Routes that mattered going forward get more of the error coming back.
• The $\circ\, \varphi'(z^{(l)})$ multiplies by how responsive the activation was at that point. Where the slope is flat (saturated), almost nothing gets through.

Once $\delta$ is in hand, the weight gradients are one product:

Here $a^{(l-1)}$ is the value from the forward pass, which is why those intermediates cannot be thrown away (section 5).

3. Checking that it is actually right

The derivation is short, and a wrong sign or a missing transpose leaves its shape intact. Implementations that run but are wrong are easy to write, so check it mechanically, with the central difference from Chapter 0-2 equation (2).

Measured: a small network of 4 inputs, hidden layers of 6 and 5, 3 outputs, loss $\tfrac{1}{2}\lVert \text{out} - y \rVert^2$, comparing equation (4) against the central difference ($h = 10^{-6}$) for every one of the 83 parameters.

They agree. The size of the difference is the central difference's own error (order $h^2$), not backpropagation's.

4. Why "backward"

As Chapter 0-2 showed, matrix products can be taken in any order, and the order changes the cost. The loss is one output; the parameters number in the millions. In that shape, multiplying from the left — descending from the output side — is the cheap one.

For the same three stages ($J_1 \in \mathbb{R}^{1\times10}$, $J_2 \in \mathbb{R}^{10\times128}$, $J_3 \in \mathbb{R}^{128\times784}$), left-to-right costs $101{,}632$ multiplications against $1{,}011{,}360$ — 9.95 times. Going left, what you hold is always a row vector (that is $\delta$), and no large matrix is ever formed.

The name "backpropagation" describes the order of computation, nothing more. The differentiation rule is the same chain rule either way; the only choice made is which end to multiply from. For a problem with one input and millions of outputs, forward would be the cheaper direction (forward-mode automatic differentiation).

5. The price — keeping every intermediate

Equation (4) contains $a^{(l-1)}$. Descending needs values computed on the way up. So during the forward pass, nothing can be discarded.

Batch 32, width 1024, 24 layers: $32 \times 1024 \times 24 \times 4\ \text{bytes} = 3.00\ \text{MB}$. Raise the batch to 256 and it is 24.00 MB (eight times). In a real model this accumulates across every kind of layer, so training memory is usually set by stored activations rather than by model size. That is why memory runs out the moment you raise the batch size.

6. Sometimes it does not arrive — vanishing gradients

Repeat equation (3) $L$ times and the $\delta$ reaching layer 1 is a product of (weight) × (activation slope) over every layer. If the per-layer factor sits below 1, depth shrinks it exponentially — the same $0.42^n$ shape seen in the Chapter 0-2 figure.

Measured at width 128 and 20 layers, with weights initialised as $\mathcal{N}(0,\, g^2/n_{\text{in}})$, calling $g$ the gain:

(The figure uses width 64 and draws its own weights, so its numbers do not match the table exactly. Measured: ReLU at $g=1$ reads $1.8\times10^{-3}$ in the table and 0.0018 in the figure; sigmoid at $g=1$, $1.4\times10^{-12}$ against $1.0\times10^{-12}$; tanh at $g=1$ is further apart, 0.933 against 0.442. What matches is the pattern, not any single measured value.)

A common misreading: "gradients vanish because of the activation function"

Half right. Look at rows two and three above. Same tanh, same 20 layers, gain moved from 1 to 2 — and $0.933$ becomes $87.7$. Not vanishing but exploding. Lower the gain instead and it vanishes.

ReLU behaves the same way: at $g=1$ it shrinks to $1.8\times10^{-3}$, and at $g=\sqrt{2}$ it returns to 1.37. So "ReLU does not vanish" is wrong too. What governs this is not the activation but the per-layer product of (weight size) × (activation slope). The $\sqrt{2}$ is exactly the factor that compensates for ReLU zeroing out half its inputs.

That said, the activation does have properties of its own. The derivative of sigmoid peaks at $0.25$, so try to recover that with gain:

It improves, but even at eight it does not reach 1. Raising the gain also saturates it — inputs grow and push $\varphi'$ into its flat region — so what you win you give back. That squeeze is why sigmoid is hard to stack deeply.

Exercises

The answers are folded away. Work them first.

Exercise 1. About equation (3). (a) What are the dimensions of $W^{(l+1)\top}$, writing $n_l$ for layer $l$'s width? (b) Why is there a transpose? (c) Why is $\circ\,\varphi'$ a Hadamard product rather than a matrix product?

Exercise 2. In a 20-layer network, $\delta$ shrinks by a factor of 0.8 per layer on average. (a) How large is layer 1's $\delta$ relative to the last layer's? (b) What about 40 layers? (c) If the per-layer factor improved to 0.95, what is the 20-layer ratio?

Exercise 3. You are checking a backward pass you wrote. (a) How? (b) How close is close enough? (c) Why can this not be used for training itself?

What this chapter does not say

• How automatic differentiation is implemented — representing the graph, registering a backward rule per operation — is not covered. Equations (3) and (4) are specific to fully connected layers; convolution and attention take other forms, derived the same way from the chain rule.
• Remedies for vanishing and exploding gradients (residual connections, normalisation layers, gradient clipping, initialisation schemes) appear by name only; they are Chapter 5 onward. The measurements here stop at what happens, not how to fix it.
• The numbers are single measurements on synthetic data with one fixed initialisation. Change the depth, width or activation and they change. What does not change is the shape: the per-layer factor acts through an exponent.
• The memory estimate counts activations only — not gradients, optimiser state (twice the weights for Adam) or temporary buffers.
• Every number was computed and checked (central-difference agreement across all 83 parameters for three activations, the per-layer $\lVert\delta\rVert$ at 20 layers, the gain sweeps for tanh and sigmoid, and the memory estimate).

Going deeper

• Backpropagation from scratch — the chain rule is all of it
• Matrix calculus from scratch — derive the backprop equations yourself
• Weight initialisation and regularisation — what makes training start, and keep going
• Neural networks from scratch — from one neuron to many layers