Read deep-learning code and almost every line is either "multiply matrices" or "multiply element by element". This chapter puts both down from the definition, states when shapes are allowed to meet, and adds two ways of looking at a matrix: eigenvalues and rank. The backpropagation of chapter 3 and the attention of chapter 8 are written in exactly the notation set here.

1. The inner product — how much two directions agree, as one number

Definition. For $x, y \in \mathbb{R}^n$ the inner product is

and $\|x\| = \sqrt{x^{\top}x}$ is the Euclidean norm of $x$.

Geometrically $x^{\top}y = \|x\|\,\|y\|\cos\theta$, where $\theta$ is the angle between the two directions. Rearranging,

which cancels the lengths and leaves agreement of direction alone. Equation (2) is cosine similarity, the quantity used to compare embeddings (chapter 10).

⛶

Worked example. For $x = (1,2,3)^{\top}$ and $y = (7,9,11)^{\top}$, $x^{\top}y = 1\cdot7 + 2\cdot9 + 3\cdot11 = 58$.

2. The matrix product — inner products, laid out in a grid

Definition. For $A \in \mathbb{R}^{m \times k}$ and $B \in \mathbb{R}^{k \times n}$, the product $AB \in \mathbb{R}^{m \times n}$ has entries

The right-hand side of equation (3) is the inner product of row $i$ of $A$ with column $j$ of $B$. So a matrix product is just $m \times n$ inner products arranged in a table. It also explains why the column count of $A$ must equal the row count of $B$: the summation index $p$ appears on both sides, so its range has to be the same or the sum cannot be written.

Worked example.

$$A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix},\quad B = \begin{pmatrix} 7 & 8 \\ 9 & 10 \\ 11 & 12 \end{pmatrix},\quad AB = \begin{pmatrix} 58 & 64 \\ 139 & 154 \end{pmatrix}$$

$(2,3) \times (3,2) \to (2,2)$: the shape contracted. Check one entry: $(AB)_{22} = 4\cdot8 + 5\cdot10 + 6\cdot12 = 32 + 50 + 72 = 154$.

Note. $AB$ and $BA$ are different in general. Above, $BA$ is a $(3,3)$ matrix — not even the same shape.

3. The Hadamard product — same shape or nothing

Definition. For $C, D \in \mathbb{R}^{m \times n}$ of equal shape, the Hadamard (element-wise) product is

Equations (3) and (4) look alike but demand opposite things of the shapes: the matrix product needs the inner dimensions to agree, the Hadamard product needs all of them to agree.

Worked example.

$$\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix} \circ \begin{pmatrix} 2 & 0 & 1 \\ 1 & 3 & 2 \end{pmatrix} = \begin{pmatrix} 2 & 0 & 3 \\ 4 & 15 & 12 \end{pmatrix}$$

The Hadamard product takes the lead in two fixed places: multiplying a gradient by the derivative of an activation (chapter 3 has $\circ$ in its equations), and gating how much to let through (the LSTM of chapter 5).

4. Rank — how many directions there really are

Definition. The rank of $A$ is the dimension of the space spanned by its columns.

When rows or columns can be written as multiples of each other, the number of genuine directions falls below the apparent size.

Worked example.

$$E = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 0 & 1 \end{pmatrix}$$

Row 2 is twice row 1. It is $3 \times 3$, yet only two rows are independent. Computing the singular values gives 8.4354, 0.9183, 0.0000 — two of them nonzero, so $\mathrm{rank}(E) = 2$ (the rank equals the number of nonzero singular values).

⛶

This view pays off when a large matrix can be replaced by a product of two small ones. Restricting $\Delta W \in \mathbb{R}^{d \times d}$ to rank $r$ lets us write $\Delta W = BA$ with $B$ of shape $d \times r$ and $A$ of shape $r \times d$, cutting the stored numbers from $d^2$ to $2dr$. That is what LoRA does (chapter 9).

5. Eigenvalues and diagonalisation — the directions a matrix cannot turn

Definition. For a square $A$, a scalar $\lambda$ with $Av = \lambda v$ for some $v \neq 0$ is an eigenvalue and $v$ an eigenvector.

An eigenvector's direction survives multiplication by $A$ unchanged; only its length scales, by $\lambda$.

⛶

Derivation. $Av = \lambda v$ is $(A - \lambda I)v = 0$. A nonzero solution exists exactly when $A - \lambda I$ has no inverse, i.e. when $\det(A - \lambda I) = 0$. Writing it out for $2 \times 2$,

that is, $\lambda^2 - (\mathrm{tr}A)\lambda + \det A = 0$.

Worked example. For $F = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ we have $\mathrm{tr}F = 4$ and $\det F = 3$, so $\lambda^2 - 4\lambda + 3 = 0$ and $\lambda_1 = 3,\ \lambda_2 = 1$, with eigenvectors $(1,1)^{\top}/\sqrt{2}$ and $(-1,1)^{\top}/\sqrt{2}$.

Diagonalisation. Put the eigenvectors in the columns of $P$ and the eigenvalues on the diagonal of $\Lambda$; then $A = P\Lambda P^{-1}$. Since $F$ is symmetric, $P$ is orthogonal and $P^{-1} = P^{\top}$. Computing $P \Lambda P^{\top}$ returns the original $\begin{pmatrix} 2&1\\1&2\end{pmatrix}$.

From here, repeated multiplication becomes cheap.

Every interior $P^{-1}P$ collapses to $I$, leaving the $k$-th power of a diagonal matrix — each eigenvalue raised to $k$. Check it: multiplying $F$ by itself four times gives $\begin{pmatrix} 41 & 40 \\ 40 & 41 \end{pmatrix}$, and $P\Lambda^4 P^{\top}$ with $\lambda^4 = 81, 1$ gives the same $\begin{pmatrix} 41 & 40 \\ 40 & 41 \end{pmatrix}$. They agree.

Equation (6) pays off in chapter 5. An RNN multiplies by the same weights once per time step, so eigenvalues above 1 make $\lambda^k$ explode and eigenvalues below 1 crush it to zero. Exploding and vanishing gradients are a statement about $\Lambda^k$ in equation (6).

6. Matching tensor shapes — nine-tenths of implementation

A "tensor" here is an array of numbers with three or more indices. What actually breaks in implementation is shape mismatch, so here is one layer, concretely.

A linear layer taking a batch of 32 inputs of 784 dimensions down to 128:

$$\underbrace{X}_{(32,\,784)} \; \underbrace{W}_{(784,\,128)} \;\to\; \underbrace{XW}_{(32,\,128)}$$

The inner 784 disappears; the outer 32 and 128 remain — exactly the shape rule of equation (3). The number of multiplications is $32 \times 784 \times 128 = 3{,}211{,}264$.

The bias $b$ has shape $(128,)$ — a single vector — yet it is added to all 32 rows. That is broadcasting: a missing axis is treated as a repetition of the same value. $(32,128) + (128,) \to (32,128)$.

What this chapter does not claim

• Diagonalisation of non-symmetric matrices is not covered. $A = P\Lambda P^{-1}$ requires a full set of $n$ eigenvectors, and some matrices do not have one (Jordan form is not used in this book).
• Singular value decomposition is used only for the fact "rank = number of nonzero singular values"; it is not derived. The definition of rank taken here is the dimension of the column space.
• Numerical error (rounding, conditioning) is not covered. The singular value printed as 0.0000 above is not exactly zero; it computes to around $10^{-16}$.

Every number in this chapter was computed and checked (matrix product, Hadamard product, singular values, eigenvalues, and the agreement of $F^4$ with $P\Lambda^4P^{\top}$).

Going deeper

• Linear algebra for AI — what vectors and matrices are actually doing
• The linear algebra behind LoRA and RAG — eigenvalues, low rank and vector search, hands on
• Tensors and shape manipulation — read einsum and you can read the papers
• Principal component analysis from scratch — eigenvectors are the axes of your data