Chapter 0-1 put down "laid out in a grid, it computes." This chapter is about what happens when you move something. Every training rule is a rule for picking one direction — the direction that makes things better — and what picks it is differentiation. There are only three symbols to carry: partial derivative, gradient, Jacobian. The chain rule is how you join them, and it has exactly two cases. Where the graph is a chain you multiply; where it branches you add.

First, the vocabulary for talking about shape.

1. Sets and maps — what "shape mismatch" means

Write $\mathbb{R}^n$ for the set of all lists of $n$ real numbers. So $x \in \mathbb{R}^{784}$ reads "$x$ is a list of 784 reals."

Definition. A rule assigning to each element of $A$ exactly one element of $B$ is a map $f: A \to B$. $A$ is the domain, $B$ the codomain.

A layer is a map. $x \mapsto Wx + b$ with $W \in \mathbb{R}^{128 \times 784}$ is a map $\mathbb{R}^{784} \to \mathbb{R}^{128}$: it takes a list of 784 and returns a list of 128.

The composite $g \circ f$ (through $f$, then through $g$) is defined only when the codomain of $f$ equals the domain of $g$. You cannot put $g: \mathbb{R}^{64} \to \mathbb{R}^{10}$ after $f: \mathbb{R}^{784} \to \mathbb{R}^{128}$.

The shape mismatch you meet every day is the machine saying exactly this. A shape that does not line up is not a failed computation; it is a composite that was never defined. "Inner dimensions must agree," from Chapter 0-1 equation (3), is this same condition written in the language of matrices.

2. Supremum and infimum — a minimum need not exist

One more piece of vocabulary before derivatives. Without it, "when do we stop?" in Chapter 4 has nothing to stand on.

Definition. A number that is $\ge$ every element of $S \subset \mathbb{R}$ is an upper bound of $S$. The least upper bound is the supremum $\sup S$. Symmetrically, the greatest lower bound is the infimum $\inf S$.

A maximum $\max S$ must both be an upper bound and belong to $S$. The supremum need not belong. That is the whole difference.

Example. $S = \{\, 1 - \tfrac{1}{n} : n = 1,2,3,\dots \,\} = \{0,\ 0.5,\ 0.6667,\ 0.75,\ 0.8,\ \dots\}$. Every element is below 1, and they get arbitrarily close to it, so $\sup S = 1$. But $1 \notin S$, so $S$ has no maximum.

This lands directly on loss functions. Take two points: $x_1 = +1$ with label $y_1 = 1$, and $x_2 = -1$ with label $y_2 = 0$, predicted by $p = \sigma(wx)$ where $\sigma(z) = 1/(1+e^{-z})$. The mean log loss works out to $L(w) = -\log\sigma(w)$.

It keeps falling. $\inf L = 0$, but $\sigma(w) < 1$ always holds, so 0 is never reached: this problem has no minimiser, and the weights grow without bound. What Chapter 1 of Part III measured — scaling a line up lowers the loss while the boundary stays put — was one face of this.

Add one contradicting point and it changes. Put label $y_3 = 1$ on $x_3 = -1$: growing $w$ now punishes that point, so the two pressures balance somewhere. Indeed $\partial L/\partial w = 0$ is equivalent to $\sigma(w) = 2/3$, giving $w = \log 2 = 0.6931\ldots$ and $L = 0.6365$. The minimum sits at a finite place.

3. Partial derivatives — move exactly one

Definition. The partial derivative of $f: \mathbb{R}^n \to \mathbb{R}$ in the $i$-th variable is

Equation (1) does one thing: hold the others still, move one. How many times more the output moved is the partial derivative.

Worked numbers. For $f(x,y) = x^2 y + 3y$ we get $\partial f/\partial x = 2xy$ and $\partial f/\partial y = x^2 + 3$. At $(x,y) = (2,3)$, $f = 21$, $\partial f/\partial x = 12$, $\partial f/\partial y = 7$.

A formula derived by hand can be checked by moving things.

Here $e_i$ has a 1 in slot $i$ and zeros elsewhere, and $h$ is small. Equation (2) is the central difference. Measuring the example above with $h = 10^{-5}$ gives $12.0000000001$ and $7.0000000000$ — at most $9.0 \times 10^{-11}$ away from the analytic values.

This check is a working tool: when you doubt a backward pass you wrote, compare it against equation (2) first. It is not for training — each parameter costs two forward passes, so 785 parameters cost 1,570 of them. It exists to verify, nothing else.

4. The gradient — stack the partials and you get the steepest direction

Definition. The gradient of $f: \mathbb{R}^n \to \mathbb{R}$ is the vector of its partials,

What makes the gradient special is that stacking alone already gives the steepest direction. That is provable.

How much $f$ moves along a unit direction $u$ is the derivative at $t=0$ of the one-variable function $\varphi(t) = f(x + tu)$. Applying the chain rule componentwise,

This is the directional derivative. Equation (4) is an inner product — the same shape as equation (1) of Chapter 0-1 — so $\nabla f^{\top} u = \|\nabla f\|\,\|u\|\cos\theta = \|\nabla f\|\cos\theta$ and therefore

The right equality holds only when $\cos\theta = 1$, i.e. $u = \nabla f/\|\nabla f\|$. The gradient direction increases $f$ fastest, and its negative decreases it fastest. The ceiling on the rate of increase is $\|\nabla f\|$ itself.

Worked numbers. At $(2,3)$ above, $\nabla f = (12,7)^{\top}$ and $\|\nabla f\| = \sqrt{193} = 13.8924$. Rotating the direction away and measuring:

Agreeing with $\|\nabla f\|\cos\theta$ to four decimals. It is 0 at 90° because that direction runs along the contour.

5. The Jacobian — the derivative of a map is a matrix

One output is served by a gradient, but a layer has 128 of them.

Definition. The Jacobian matrix of $f: \mathbb{R}^n \to \mathbb{R}^m$ is

The shape need not be memorised: rows count outputs, columns count inputs. With $m = 1$ it is $(1,n)$ — the gradient laid on its side.

In deep learning two cases cover almost everything.

(a) A linear layer. For $z = Wx$ with $W \in \mathbb{R}^{128 \times 784}$, $\partial z_i/\partial x_j = W_{ij}$, so $J = W$. The weights are the Jacobian.

(b) An elementwise activation. If $a = \varphi(z)$ applies componentwise, $a_i = \varphi(z_i)$, then $\partial a_i/\partial z_j = 0$ whenever $i \neq j$, so

$$J = \mathrm{diag}\big(\varphi'(z_1), \dots, \varphi'(z_{128})\big)$$

a diagonal matrix. This decides the shape of the code. Multiplying by a diagonal matrix is the same as multiplying elementwise, which is why the equations in Chapter 3 carry a Hadamard product $\circ$ (Chapter 0-1, equation (4)) instead of a matrix product. The storage differs too: as a $(128,128)$ matrix it is $16{,}384$ numbers; as a diagonal, $128$ — a factor of 128.

(c) A general example. For $g(x,y) = (x^2,\ x+y)$,

$$J_g = \begin{pmatrix} 2x & 0 \\ 1 & 1 \end{pmatrix}, \qquad J_g(2,3) = \begin{pmatrix} 4 & 0 \\ 1 & 1 \end{pmatrix}$$

Measuring all four entries with the central difference of equation (2), the largest gap was $3.8 \times 10^{-11}$.

6. The chain rule — multiply along a chain, add across a branch

In one variable, $(h \circ g)'(x) = h'(g(x))\, g'(x)$: a product of local slopes. For maps the shape is identical, with the product becoming a matrix product.

Check the shapes in equation (7). With $g: \mathbb{R}^n \to \mathbb{R}^k$ and $f: \mathbb{R}^k \to \mathbb{R}^m$, the right side is $(m,k) \times (k,n) \to (m,n)$; the left is the derivative of a map $\mathbb{R}^n \to \mathbb{R}^m$, so $(m,n)$. They agree — because the condition for a composite to exist (section 1) and the condition for a matrix product to exist (Chapter 0-1, equation (3)) were the same condition all along.

Written in components, you can see where the addition comes from.

The sum in equation (8) is a sum over paths. There are $k$ routes from $x_j$ to $(f\circ g)_i$, one per intermediate $u_p$. Multiply the local slopes along each route, then add the routes up. Chain: multiply. Branch: add. That is the entire rule.

Numerical check. Put $f(u,v) = uv$ after $g(x,y) = (x^2,\ x+y)$; the composite is $h(x,y) = x^2(x+y)$. At $(2,3)$,

$$J_f = (v,\ u) = (5,\ 4), \qquad J_f J_g = (5,\ 4)\begin{pmatrix} 4 & 0 \\ 1 & 1 \end{pmatrix} = (24,\ 4)$$

Differentiating directly: $\partial h/\partial x = 3x^2 + 2xy = 24$ and $\partial h/\partial y = x^2 = 4$. They match, and the gap against the central difference was $1.5\times10^{-10}$.

The order you multiply in sets the cost

Chain equation (7) across the layers and you get a run of matrix products. The order is yours to choose (matrix products are associative) — and the choice is worth a factor of ten.

Take three stages going from the loss (one output) back toward the input: $J_1 \in \mathbb{R}^{1 \times 10}$, $J_2 \in \mathbb{R}^{10 \times 128}$, $J_3 \in \mathbb{R}^{128 \times 784}$.

9.95 times. Multiplying from the left, what sits in your hands is always a row vector; a $(128,784)$ matrix is never formed. Multiplying from the right, it is.

A common misreading: "follow the gradient and you take the shortest path to the valley"

Section 4 proved "steepest at this point." Nothing there says anything about the length of the journey. Measure it and the difference shows up plainly.

Take $f(x,y) = (x^2 + 10y^2)/2$, whose valley floor is the origin, and descend from $(10,1)$ with $\eta = 0.18$.

The path is 1.34× long because of the sign of the $y$ component. The first six values are $-0.8000$, $+0.6400$, $-0.5120$, $+0.4096$, $-0.3277$, $+0.2621$ — it crosses over on every step. Curvature along $y$ is ten times tighter, so a step sized for $x$ must overshoot in $y$. It zigzags across the valley while creeping toward the centre.

Line the curvature up across axes and the gap disappears. From the same start, $f(x,y) = (x^2+y^2)/2$ gives a path length of 10.0402 — the straight-line 10.0499 minus the 0.01 radius where we stopped. Same 35 steps. What created the detour was not the gradient but the difference in curvature between axes.

Exercises

The answers are folded away. Work them first.

Exercise 1. Let $W \in \mathbb{R}^{128 \times 784}$, $z = Wx$, and $a = \tanh(z)$ componentwise. (a) Shape and contents of the Jacobian of $x \mapsto z$. (b) Shape and contents for $z \mapsto a$. (c) Shape for $x \mapsto a$. (d) Why does the code never hold (b) as a matrix?

Exercise 2. Consider $S = \{\, -\log\sigma(z) : z \in \mathbb{R} \,\}$. (a) What is $\inf S$, and does it belong to $S$? (b) Is $S$ bounded above? (c) How does the answer to (a) look in float64?

Exercise 3. Split $y = (x+1)x^2$ into the two routes $u = x+1$ and $v = x^2$. At $x = 2$: (a) the product along each route and their sum; (b) the value by direct differentiation; (c) what do you get counting only the upper route, and what has gone wrong?

What this chapter does not say

• Points where the derivative does not exist are left out. ReLU has no slope at $z=0$. Implementations pick one subgradient (usually 0) and move on; that choice is not justified here.
• Second derivatives (the Hessian) are not used. We measured that differing curvature makes training harder, but handling curvature itself as a matrix (second-order methods) is only touched on in Chapter 4.
• The rigorous ($\varepsilon$-$\delta$) definitions of limit and continuity are not laid down. Supremum and infimum were brought in for one purpose: to say that a minimum need not exist.
• How automatic differentiation is implemented — what the graph holds and how much memory it costs — is Chapter 3. This chapter is only the rule.
• Every number in the text was computed and checked (central-difference agreement for the partials and the Jacobian, the directional derivative against $\|\nabla f\|\cos\theta$, $J_fJ_g$ against direct differentiation, the $w = \log 2$ minimum, the descent path length and step count, and the float64 rounding threshold).

Going deeper

• Calculus for AI — the gradient is an arrow saying which way gets better
• Matrix calculus from scratch — derive the backprop equations yourself
• Backpropagation from scratch — the chain rule is all of it
• Gradient descent from scratch — the geometry of learning
• Convexity and optimisation — why deep learning works although it is not convex