Linear Algebra for AI — What Vectors and Matrices Are Actually Doing

You will never compute a determinant to read an AI paper. What you do need is two ideas: a vector is a coordinate where meaning lives, and a matrix is a machine that carries coordinates somewhere else. From why a dot product means similarity to reading the attention equation symbol by symbol.

The goal: being able to read this equation

Open almost any modern AI paper and something like this shows up by page two.

In words: $Q$, $K$ and $V$ are not operations but three bundles of numbers — roughly what each word is asking about, what each word advertises about itself, and what each word actually carries. Everything else on the line is arithmetic performed on those three bundles.

What this equation says, stripped of notation, is: decide how much each word should look at each other word using dot products, then blend the information in those proportions. If it means nothing yet, that is fine. We come back to it at the end and name every symbol.

There are only two tools inside it. A vector is a coordinate where meaning is placed, and a matrix is a transformation that carries coordinates into a different space. Determinants and Cramer's rule essentially never appear in this literature. Of everything in a university linear algebra course, a small slice is what you actually use.

Vectors: meaning as coordinates

A vector is a list of numbers. Write $\mathbf{v} = [0.3, -1.2, 0.8]$ and you have three numbers, read as an arrow pointing at one location in three-dimensional space. The bold type is nothing more than a reminder that this is a vector.

The reason this matters for AI is that meaning can be placed in such a space. Assign a few hundred numbers to "dog", assign another few hundred to "cat". If the assignment is any good, dog and cat land near each other, while dog and "accounts payable" land far apart. That assignment is an embedding.

One thing to be clear about: the individual axes do not carry human-readable meanings like "axis 1 is number of legs". Meaning lives not in the axes but in the relative positions of vectors. Which is exactly why the next tool you need is a way to measure how close two of them are.

The dot product: turning "same direction?" into one number

In words: walk down the two lists side by side, multiply each pair of matching entries, add all of those products together — and that one total is the same number as how long the first arrow is, times how long the second one is, times how far the two are from pointing the same way.

Symbol by symbol: $d$ is the number of dimensions (how many numbers are in the list), $a_i$ is the $i$-th component of $\mathbf{a}$, $\sum$ says "add all of these up", $\|\mathbf{a}\|$ is the length of the arrow, and $\theta$ ("theta") is the angle between the two arrows.

What this equation is really saying is that a dumb bookkeeping operation — multiply matching entries, add them all — turns out to equal "length times length times how well the directions agree". The left side is arithmetic a machine can grind through; the right side is geometry. That those two are the same object is very nearly the whole reason linear algebra earns its place in AI.

To ignore length and compare direction alone, divide by both lengths.

$$\mathrm{sim}(\mathbf{a},\mathbf{b}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|} = \cos\theta$$

In words: the top of the fraction is the raw dot product, the bottom is the two lengths multiplied together, so the lengths cancel out and only the angle survives the division.

In plain terms: compare only which way the two arrows point. Same direction gives 1, perpendicular gives 0, opposite gives −1. This is cosine similarity, and it is what "retrieve the nearest documents" actually means in RAG Fundamentals. Vector search is nothing more exotic than sorting by dot product with the query vector.

Matrices: machines that carry vectors elsewhere

A matrix is a grid of numbers, but the grid is not the point. A matrix is a device that takes a vector in and hands a different vector back — a transformation.

In words: $W$ is the machine, $\mathbf{x}$ is what you feed into it, $\mathbf{y}$ is what comes out, and the sum on the right is the recipe for one single entry of that output rather than for the whole thing at once.

Here $\mathbf{x}$ is the input vector (length $d_{\text{in}}$), $\mathbf{y}$ is the output vector (length $d_{\text{out}}$), and $W_{ij}$ is the number in row $i$, column $j$ of $W$.

What this says is: the $i$-th output is the dot product of the whole input with row $i$ of $W$. Each row of $W$ holds one question; the input is asked every question in turn, and the answers stacked up form the output. A fully connected layer is precisely this, and the number of rows of $W$ is the number of neurons in that layer (Neural Networks From Scratch).

A word about shape. If $W$ is $d_{\text{out}} \times d_{\text{in}}$ and $\mathbf{x}$ has $d_{\text{in}}$ entries, the result has $d_{\text{out}}$ entries; the inner numbers must match or nothing multiplies. Most implementation errors live here — and when reading a paper, simply tracking shapes tells you what is being turned into what.

Matrix multiplication is a table of every dot product

$$(AB)_{ij} = \sum_{k} A_{ik}B_{kj}$$

In words: the subscript $ij$ picks out one single cell of the answer table, and $k$ is the counter you sweep along to fill in that one cell — everything on the right is the work behind a single number.

Read it as: take the dot product of row $i$ of $A$ with column $j$ of $B$ and write it at position $(i,j)$. Matrix multiplication is a table of equation (2) evaluated for every pair, and nothing more.

With that, $QK^{\top}$ from the opening becomes legible. If $Q$ stacks up the query vectors and $K$ stacks up the key vectors, then $QK^{\top}$ is every query dotted with every key, laid out in one table. Entry $(i,j)$ is the raw score for "how much should word $i$ look at word $j$".

Keep a sense of cost too. Multiplying an $m \times k$ matrix by a $k \times n$ one takes roughly $2mkn$ multiply-and-add operations. Nearly all the time spent in training and inference disappears into that product. The reason specialised hardware exists at all is that building these exhaustive tables is brutally expensive.

What a dimension actually is

A dimension is an axis — equivalently, one slot in the list of numbers. A 768-dimensional embedding represents a single word with 768 numbers.

Intuition fails in high dimensions. If you keep one property, keep this: the higher the dimension, the closer to perpendicular two unrelated vectors tend to be. Their dot product sits near zero, so "unrelated" is the default state. That is why hundreds of thousands of embeddings can share one space without smearing into each other. Adding dimensions buys room for meanings to pass one another without colliding.

It also costs. Embedding size is chosen as a trade between expressive room on one side and memory and compute on the other.

Eigenvalues and singular values: summarising a matrix's character

A matrix is a transformation. Can we say briefly what kind of transformation it is? That is what eigenvalues and singular values are for.

In words: the left-hand side means "run this arrow through the machine", the right-hand side means "just make the same arrow longer or shorter", and the equals sign is the claim that for certain arrows those two do the identical thing.

$\lambda$ ("lambda") is a number and $\mathbf{v}$ is a non-zero vector. What this says is: there exist special directions that this transformation does not rotate at all — it merely stretches them by a factor of $\lambda$. Those directions are eigenvectors, the factors are eigenvalues, and together they are the skeleton of the transformation.

Eigenvalues only exist for square matrices. The version that works for any shape is the singular value decomposition (SVD).

$$A = U\Sigma V^{\top}$$

In words: any grid of numbers $A$, however lopsided, can be rewritten as three simpler grids multiplied together — two that only turn things around ($U$ and $V^{\top}$) and one in the middle that only resizes them ($\Sigma$).

$\Sigma$ ("Sigma") has numbers only on its diagonal — those are the singular values — while $U$ and $V$ do the rotating. The message is: every matrix whatsoever factors into rotate, then stretch or shrink along each axis, then rotate again. The singular values are those stretch factors, listed largest first.

One practical consequence matters. If only a handful of singular values are large, the matrix's information is concentrated in a handful of directions — the matrix is said to be low rank. That is where the trick of replacing an enormous matrix with the product of two small ones comes from. LoRA is a bet on exactly this: that the weight update during fine-tuning is low rank enough to get away with it.

Back to the opening equation

Equation (1), once more.

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

In words, one clause at a time: score, shrink the scores, turn them into proportions, blend. The paragraph below is that same reading with the symbols named.

You can read it now. $Q$, $K$ and $V$ are bundles of query, key and value vectors, each made by multiplying the input embeddings by a different matrix. $QK^{\top}$ is the all-pairs dot product table — the raw "who looks at whom" scores. Dividing by $\sqrt{d_k}$ (where $d_k$ is the key dimension) is there because dot products grow with dimension, and feeding large numbers straight into softmax collapses the distribution onto a single token. Softmax turns each row of the table into proportions summing to one, and multiplying by $V$ blends the values in those proportions.

So the sentence behind the symbols is: use dot products to decide where to look, then average the content accordingly. In linear algebra terms, nothing but dot products and matrix multiplication. The mechanism itself is covered in Attention From Scratch.

Three things that matter in practice

1. Track shapes and the equation reads itself Stuck on an equation? Write down the shape of every symbol: $d_{\text{model}}$, $d_k$, sequence length, batch size. There is usually exactly one reading in which the shapes line up, and that reading is the right one.

2. Normalise before using dot products as similarity Skip normalisation and irrelevant documents rank highly purely for having long vectors. This is the classic explanation for a vector search that returns nonsense.

3. When you see "low rank", read "savings" LoRA, embedding compression, cache reduction — different vocabulary, one idea: approximate a big matrix by the product of two small ones.

Summary

• A vector is a coordinate where meaning is placed; the meaning is in relative position, not in individual axes
• A dot product is length times length times directional agreement; divide out the lengths and you compare direction alone — cosine similarity
• A matrix carries vectors into another space; each row asks one question and the output is the list of answers
• Matrix multiplication is a table of all pairwise dot products, and it consumes most of AI's compute
• Eigenvalues and singular values summarise a transformation's character; concentrated singular values make low-rank approximation pay

Next comes the arrow that tells you which way to move to get better: Calculus for AI. The procedure that actually does the moving is in Loss Functions and Optimization.