Backpropagation from Scratch — It Is All Just the Chain Rule

Why you can get gradients for ten million parameters for roughly the cost of one forward pass. The chain rule, computational graphs, a two-layer network worked by hand with real numbers, and where vanishing gradients come from — every symbol explained as it appears.

The soup is too salty. Which ingredient is to blame?

Your soup came out too salty. Was it the salt, the soy sauce, or reducing it too long? What you actually want to know is: how much would cutting each one back change the saltiness?

The naive approach is to remake the soup with slightly less salt and taste the difference, then remake it again with slightly less soy sauce. Twenty ingredients means twenty remakes. Written as math, that is numerical differentiation.

$$\frac{\partial L}{\partial \theta_i} \approx \frac{L(\theta + \varepsilon e_i) - L(\theta)}{\varepsilon}$$

One symbol at a time: $L$ is the loss, a single number measuring how wrong the model is. $\theta$ is every parameter bundled together, and $\theta_i$ is the $i$-th one. $\varepsilon$ (epsilon) is a tiny number, and $e_i$ is a vector that is 1 in position $i$ and 0 everywhere else. The $\partial$ ("partial") symbol means differentiating with respect to one variable while holding the rest fixed.

What this says, in plain words: nudge exactly one knob and measure how much the loss moved. It is correct, and it is hopelessly slow. Each parameter costs one forward pass (running an input through the model to get an output), so ten million parameters means ten million forward passes per training step. Training would never finish.

Backpropagation walks the recipe backwards instead, producing gradients for every parameter at once for roughly the cost of a single forward pass. There is no magic in it — just the chain rule from high school calculus, applied relentlessly.

This article assumes the neural network basics: that a neuron computes $\sigma(w^\top x + b)$, and that the forward pass is a chain of matrix products.

The chain rule — derivatives are ratios, and ratios multiply

The chain rule is one line.

$$\frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}$$

We have a three-stage relationship: changing $x$ changes $y$, and changing $y$ changes $z$.

What it says, in plain words: conversion ratios chain by multiplication. If moving $x$ by 1 moves $y$ by 3, and moving $y$ by 1 moves $z$ by 2, then moving $x$ by 1 moves $z$ by 6. It is the same arithmetic as gear ratios in a gear train.

That is all a derivative is: a local ratio. Nudge the input a hair at some point, and the output moves this many times as much. A deep network is nothing more than dozens of these gears meshed in series.

The computational graph — values forward, gradients backward

View the network as a computational graph: operations are nodes, values flow along edges. The multiplication $w_1 x$ is one node, the addition of $b_1$ is the next, ReLU is the next.

On that graph, backpropagation becomes a remarkably mechanical chore. The only thing a node needs to know is its local derivative — how much its own output moves when its own input is nudged. After that, every node just computes

incoming gradient from upstream × own local derivative = gradient passed downstream

and hands the result to its neighbour, repeating from the output end toward the input end. No node ever sees the whole network, and nobody has to expand a monstrous formula by hand. That every node can be this shortsighted and still produce globally correct gradients is the beautiful part.

There is one catch. Computing a local derivative requires the values from the forward pass, so the intermediate values have to be kept, not discarded. That is precisely why training eats so much memory.

The archetypal local derivative is the slope of an activation function. Check the magnitude of that slope in the figure below — the number you read off there is the number that gets multiplied in during backpropagation.

A two-layer network, worked by hand with real numbers

Words only get you so far, so let us turn the crank on the smallest possible network: one input, one hidden ReLU unit, one output. Four parameters total.

Start with $x = 2.0$, $w_1 = 0.5$, $b_1 = 0$, $w_2 = 1.5$, $b_2 = 0$, and a target of $y = 0.5$. The loss is squared error, $L = \tfrac{1}{2}(\hat{y} - y)^2$ — the $\tfrac{1}{2}$ is a convention that cancels the factor of 2 when you differentiate.

Forward pass (recording every value as we go)

• $z_1 = w_1 x + b_1 = 0.5 \times 2.0 + 0 = 1.0$
• $a_1 = \mathrm{ReLU}(z_1) = 1.0$
• $\hat{y} = w_2 a_1 + b_2 = 1.5 \times 1.0 + 0 = 1.5$
• $L = \tfrac{1}{2}(1.5 - 0.5)^2 = 0.5$

Here $z$ is the value before the activation and $a$ is the value after it. That distinction matters on the way back.

Backward pass (starting at the output)

• $\dfrac{\partial L}{\partial \hat{y}} = \hat{y} - y = 1.0$ — differentiating the squared error gives exactly this
• The output layer has no activation, so $\dfrac{\partial L}{\partial w_2} = 1.0 \times a_1 = 1.0$ and $\dfrac{\partial L}{\partial b_2} = 1.0$
• $\dfrac{\partial L}{\partial a_1} = 1.0 \times w_2 = 1.5$ — blame is handed to the hidden layer in proportion to the weight
• $\dfrac{\partial L}{\partial z_1} = 1.5 \times \mathrm{ReLU}'(z_1) = 1.5 \times 1 = 1.5$ — the slope is 1 because $z_1 = 1.0 > 0$
• $\dfrac{\partial L}{\partial w_1} = 1.5 \times x = 3.0$ and $\dfrac{\partial L}{\partial b_1} = 1.5$

Notice the shape of every weight gradient: upstream gradient × the value that fed into that weight. $w_2$ gets multiplied by $a_1$, $w_1$ by $x$. The more strongly an input was used, the more responsibility its weight carries — which is exactly what intuition demands.

The update

$$\theta \leftarrow \theta - \eta \frac{\partial L}{\partial \theta}$$

Here $\theta$ is any parameter, $\eta$ (eta) is the learning rate, and $\leftarrow$ means assignment. What it says, in plain words: step opposite to the direction that increases the loss, by an amount proportional to the gradient. With $\eta = 0.1$ we get $w_1 = 0.2$, $b_1 = -0.15$, $w_2 = 1.4$, $b_2 = -0.1$.

Run the forward pass again: $z_1 = 0.25$, $a_1 = 0.25$, $\hat{y} = 0.25$, and the loss is $0.03125$, down from $0.5$ in a single step. But look closer — the prediction jumped from $1.5$ (above the target) to $0.25$ (below it). The error flipped sign. That overshoot is the seed of what goes wrong when the learning rate is too high.

Generalising to any number of layers

Write the same procedure in matrix form and it extends to arbitrary depth.

$$\delta^{(l)} = \left( W^{(l+1)\top} \delta^{(l+1)} \right) \odot \sigma'(z^{(l)})$$

$$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} \left( a^{(l-1)} \right)^\top$$

The symbols: $\delta^{(l)}$ (delta) is layer $l$'s sensitivity of the loss to its pre-activation $z^{(l)}$ — the general form of the $\partial L / \partial z_1$ we just computed by hand. $W^{(l+1)\top}$ is the transpose of the next layer's weight matrix (rows and columns swapped). $\odot$ is the element-wise product, multiplying matching positions rather than doing a matrix product. And $a^{(l-1)}$ is the previous layer's output.

The first line says, in plain words: redistribute the next layer's blame through the weight matrix, then multiply by your own activation's slope. The second says: a weight's gradient is its layer's blame times the value that fed into it. Same content as the hand calculation, different notation.

Why going backwards is the efficient direction

The key fact is that the loss is a scalar — a single number.

Travelling back from the output, every node acquires exactly one sensitivity: how much the loss moves if this value moves. Compute that single number once and you can reuse it for every parameter hanging off that node. In the hand calculation, $\partial L / \partial z_1 = 1.5$ was computed once and then served both $w_1$'s gradient and $b_1$'s. The deeper the network, the more this sharing pays.

Going forward instead means tracking each parameter's influence separately, which needs as many passes as there are parameters — the same reason numerical differentiation was hopeless.

When there is one output and an enormous number of inputs, working backwards is dramatically cheaper. Backpropagation is universal in machine learning precisely because the loss is always one scalar.

Vanishing gradients fall out of the multiplication

Because the chain rule multiplies, one side effect is unavoidable. A one-unit-per-layer chain makes it obvious.

$$\frac{\partial L}{\partial z^{(1)}} = \frac{\partial L}{\partial z^{(L)}} \prod_{l=2}^{L} w^{(l)} \sigma'\!\left(z^{(l-1)}\right)$$

$\prod$ (capital pi) means "multiply all of these together," and $L$ is the number of layers. What it says, in plain words: the gradient that reaches the earliest layer is the product of every "weight × activation slope" along the way.

Products behave violently. If each factor is below 1 the result collapses exponentially toward zero (vanishing gradients); if each is above 1 it explodes exponentially (exploding gradients). As we saw last time, the sigmoid derivative never exceeds 0.25, so ten layers give $0.25^{10}$ — roughly one in a million. The early layers barely learn at all.

Every remedy targets the same thing: protecting that chain of multiplications. ReLU sets the positive-side slope to 1 so nothing decays. Residual connections open a path where the gradient passes through multiplied by 1. Normalisation layers keep each layer's input scale consistent so no factor gets extreme. On the exploding side, gradient clipping caps the magnitude.

Backpropagation in numpy

The calculation we just did by hand translates directly.

def backward(x, y, w1, b1, w2, b2):
    z1 = w1 * x + b1
    a1 = np.maximum(0, z1)        # ReLU
    y_hat = w2 * a1 + b2          # forward pass: keep the intermediates

    d_yhat = y_hat - y            # dL/dy_hat
    d_w2, d_b2 = d_yhat * a1, d_yhat
    d_a1 = d_yhat * w2            # redistribute blame through the weight
    d_z1 = d_a1 * (z1 > 0)        # multiply by the activation slope (ReLU')
    d_w1, d_b1 = d_z1 * x, d_z1
    return d_w1, d_b1, d_w2, d_b2

(z1 > 0) is the derivative of ReLU: booleans act as 1 and 0, so a positive input contributes a slope of 1 and a negative one contributes 0. The code makes both key facts visible at a glance — the backward pass mirrors the forward pass, and it reuses the forward values a1 and z1.

Three things that matter in practice

1. If you implement it yourself, gradient-check it. Compare your analytic gradients against numerical differentiation. In double precision the relative error should be tiny; if the magnitudes disagree, you have a bug. Backprop bugs tend to surface as "training is a bit slow" rather than an outright crash, so they are worth catching the moment you write the code.

2. Training memory is mostly stored intermediates. The backward pass needs the forward pass's intermediate values, so doubling the batch size roughly doubles the activations you must keep. This is the usual reason inference fits but training runs out of memory. Gradient checkpointing discards some intermediates and recomputes them on demand, trading compute for memory.

3. When you see NaN, suspect the forward pass before the backward pass. NaN gradients usually originate upstream: $\log(0)$, a division by zero, or values that diverged because the learning rate was too large. Gradient clipping suppresses the symptom without removing the cause.

Summary

• Backpropagation is the chain rule applied mechanically; each node only needs its own local derivative
• Passing "upstream gradient × local derivative" from output to input yields every parameter's gradient in one sweep
• Backwards is fast because the loss is a single scalar, so partial results can be shared on the way back
• Because it is a chain of products, factors below 1 vanish and factors above 1 explode — ReLU, residual connections, and normalisation all exist to protect that chain

The autograd in PyTorch or JAX is doing nothing more than generating this procedure automatically from the graph. Read detach() and no_grad() as "cut the chain here" and their behaviour stops being mysterious. Next, move on to what gets built on this foundation: attention.