Matrix Calculus from Scratch — Derive the Backward Pass Yourself

Where does the transpose in ∂L/∂W = XᵀG actually come from? Matrix calculus is not a formula sheet to memorize — it is one move: rotate dX to the right inside a trace. From denominator layout and shape-checking to the gradients of a linear layer and softmax + cross-entropy, ending with a double-precision gradient check.

When the knobs become a table, so does the derivative

The story about tasting a soup that came out too salty and hunting for the guilty ingredient is in Backpropagation from Scratch. There we chased a handful of knobs. A real network has hundreds of thousands of them in a single layer — and they are not a flat list of numbers. They sit in a table with rows and columns: a matrix.

The calculus you were taught in school was about functions that take a number and return a number. What deep learning asks you to differentiate is a function that takes a table and returns a single number, the loss. Differentiation in that shape is called matrix calculus.

The goal is not to memorize a table of identities. The goal is to be able to derive a backward pass yourself, and check it yourself. As long as $\partial L/\partial W = X^\top G$ is something you simply accept, you stall the moment your own layer returns a gradient whose shape does not line up.

A gradient is a map with the same shape

Start with the single most useful fact: when you differentiate a scalar loss with respect to anything, the result has the same shape as that thing.

If $W$ is $3\times4$, then $\partial L / \partial W$ is also $3\times 4$, and its $(i,j)$ entry is

which says, in words: "if I nudge the knob in row $i$, column $j$ of $W$, how fast does the loss $L$ move?" That is an ordinary partial derivative. All we did was lay the twelve of them out in the same seating chart as the knobs themselves. A $3\times4$ gradient is twelve partial derivatives arranged in the original layout. No new concept has entered the room.

So where is the difficulty? In the fact that computing them one at a time is hopeless. If $W$ is $4096\times4096$, that is sixteen million entries. You need a way to write the whole thing down at once, matrix-shaped. That is matrix calculus.

Denominator layout — why transposes keep flipping

Open two textbooks and the same quantity is defined transposed in each. This is the single biggest time sink in matrix calculus, and the cause is not deep mathematics. There are simply two conventions for arranging the entries.

Take a vector $y$ (with $m$ components) differentiated with respect to a vector $x$ (with $n$ components). The contents are the $m\times n$ partial derivatives $\partial y_i / \partial x_j$, identical under either convention. Only the arrangement differs.

• Numerator layout: the numerator's index ($y$) runs down the rows. Shape $m \times n$. This is what people normally call the Jacobian.
• Denominator layout: the denominator's index ($x$) runs down the rows. Shape $n \times m$ — the transpose of the above.

Deep learning uses a hybrid. Scalar-with-respect-to-anything uses denominator layout (a gradient has the shape of the thing it differentiates), while an explicitly written Jacobian uses numerator layout. Mixing them is safe because the only thing a framework ever hands back to you is the gradient of a scalar loss; Jacobians are intermediate multiplication material.

And there is exactly one thing to remember: whichever one makes the shapes work is the right one. If you cannot recall whether it was $X^\top G$ or $G^\top X$, pick the one that comes out shaped like $W$ and you will be right. Trying to memorize transpose directions is the least rewarding effort in this whole subject.

There is only one pattern to learn

Deriving these by writing out $\sum_k$ index by index is possible, but you will make a mistake as soon as three indices are in play. What practitioners actually use is the differential, and the pattern is this. Once you can rearrange the small change in the loss into the form

you can read off $\partial L/\partial X = G$. Here $\mathrm{d}X$ is "the way you nudged $X$" — a matrix the same shape as $X$ — and $\mathrm{tr}$ is the trace, the sum of the diagonal entries. In words: if you can write the wobble in the loss as an inner product with the nudge, whatever sits on the other side of that inner product is the gradient.

Why a trace, of all things? Because $\mathrm{tr}(G^\top X)$ written out entry by entry is $\sum_{i,j} G_{ij}X_{ij}$ — the inner product of two tables flattened into long vectors. The one-variable statement $\mathrm{d}L = g\,\mathrm{d}x$ (slope times nudge) lifted to many components turns that multiplication into an inner product, and the trace is just how you write an inner product of matrices. Nothing to brace yourself for.