Backpropagation from Scratch — It Is All Just the Chain Rule

Why you can get gradients for ten million parameters for roughly the cost of one forward pass. The chain rule, computational graphs, a two-layer network worked by hand with real numbers, and where vanishing gradients come from — every symbol explained as it appears.

The soup is too salty. Which ingredient is to blame?

Your soup came out too salty. Was it the salt, the soy sauce, or reducing it too long? What you actually want to know is: how much would cutting each one back change the saltiness?

The naive approach is to remake the soup with slightly less salt and taste the difference, then remake it again with slightly less soy sauce. Twenty ingredients means twenty remakes. Written as math, that is numerical differentiation.

$$\frac{\partial L}{\partial \theta_i} \approx \frac{L(\theta + \varepsilon e_i) - L(\theta)}{\varepsilon}$$

One symbol at a time: $L$ is the loss, a single number measuring how wrong the model is. $\theta$ is every parameter bundled together, and $\theta_i$ is the $i$-th one. $\varepsilon$ (epsilon) is a tiny number, and $e_i$ is a vector that is 1 in position $i$ and 0 everywhere else. The $\partial$ ("partial") symbol means differentiating with respect to one variable while holding the rest fixed.

What this says, in plain words: nudge exactly one knob and measure how much the loss moved. It is correct, and it is hopelessly slow. Each parameter costs one forward pass (running an input through the model to get an output), so ten million parameters means ten million forward passes per training step. Training would never finish.

Backpropagation walks the recipe backwards instead, producing gradients for every parameter at once for roughly the cost of a single forward pass. There is no magic in it — just the chain rule from high school calculus, applied relentlessly.

This article assumes the neural network basics: that a neuron computes $\sigma(w^\top x + b)$, and that the forward pass is a chain of matrix products.

The chain rule — derivatives are ratios, and ratios multiply

The chain rule is one line.

$$\frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}$$

We have a three-stage relationship: changing $x$ changes $y$, and changing $y$ changes $z$.

What it says, in plain words: conversion ratios chain by multiplication. If moving $x$ by 1 moves $y$ by 3, and moving $y$ by 1 moves $z$ by 2, then moving $x$ by 1 moves $z$ by 6. It is the same arithmetic as gear ratios in a gear train.

That is all a derivative is: a local ratio. Nudge the input a hair at some point, and the output moves this many times as much. A deep network is nothing more than dozens of these gears meshed in series.

The computational graph — values forward, gradients backward

View the network as a computational graph: operations are nodes, values flow along edges. The multiplication $w_1 x$ is one node, the addition of $b_1$ is the next, ReLU is the next.

On that graph, backpropagation becomes a remarkably mechanical chore. The only thing a node needs to know is its local derivative — how much its own output moves when its own input is nudged. After that, every node just computes

incoming gradient from upstream × own local derivative = gradient passed downstream

and hands the result to its neighbour, repeating from the output end toward the input end. No node ever sees the whole network, and nobody has to expand a monstrous formula by hand. That every node can be this shortsighted and still produce globally correct gradients is the beautiful part.

There is one catch. Computing a local derivative requires the values from the forward pass, so the intermediate values have to be kept, not discarded. That is precisely why training eats so much memory.

The archetypal local derivative is the slope of an activation function. Check the magnitude of that slope in the figure below — the number you read off there is the number that gets multiplied in during backpropagation.