2026-08-21·FREE·7 min read

The Linear Algebra Under LoRA and RAG — Eigenvalues, Low Rank and Vector Search, Hands On

A matrix is a deformation of space, an eigenvector is a direction that survives it, SVD generalises the idea, and the dot product is the definition of 'similar'. Four interactive figures and four equations show that LoRA's ΔW=BA and RAG's vector search stand on the same floor. A column meant to be dragged, not just read.

#linear-algebra #eigenvalues #svd #lora #rag #vector-search #math

ModalitytextTaskmath

A matrix is not a table — it is a deformation

The single biggest obstacle in linear algebra is reading a matrix as a table of numbers. Read it that way and eigenvalues become a recipe you memorise.

A matrix $A$ is a machine that deforms space. Feed it a vector $v$ and a different vector $Av$ comes out. In 2×2 it deforms the plane: feed it the unit circle and an ellipse comes back. Once you switch to this view, LoRA and RAG turn out to sit in the same landscape.

Eigenvectors — directions the deformation cannot turn

Most vectors change direction when pushed through $A$ . A few special ones stay on their own line: stretched or squashed, but not rotated. Those are the eigenvectors; the stretch factor is the eigenvalue.

A\boldsymbol{v} = \lambda \boldsymbol{v}

(1)

The equation, in words: find a direction $v$ that $A$ merely scales by $\lambda$ , without turning it.

For a 2×2 matrix you can solve it by hand. Expanding $\det(A - \lambda I) = 0$ gives

\lambda^2 - (a+d)\,\lambda + (ad - bc) = 0

(2)

— an ordinary quadratic, which says the eigenvalues are settled by just two numbers: the trace $a+d$ and the determinant $ad-bc$ . And when the discriminant goes negative there is no real solution: a map with rotation mixed in has no real direction it leaves unturned. The quadratic tells you that directly.

Try it below. The two dashed lines are the eigenvector directions. Rotate $v$ until $Av$ lands on the same line. Pick the "rotation" preset and the dashed lines vanish — no real eigenvectors exist.

FIG 1A 2×2 deformation. The faint ring of dots is where the unit circle lands. Dashed lines are eigenvector directions — align v with one and Av stays on that line, merely scaled by λ

SVD — every matrix is rotate, stretch, rotate

Eigenvalues only apply to square matrices. The singular value decomposition extends the idea to any shape:

A = U\Sigma V^{\top}

(3)

Any matrix factors into "rotate ( $V^\top$ ) → stretch along axes ( $\Sigma$ ) → rotate ( $U$ )". The stretch factors $\sigma_1 \ge \sigma_2 \ge \dots$ on the diagonal of $\Sigma$ are the singular values — the strengths of the directions along which the matrix carries its information.

What the factorisation amounts to, in words: however tangled a deformation looks, all it ever does is line the axes up, stretch along them, and put them back. $U$ and $V$ change direction without changing length; $\Sigma$ alone does the stretching. So the character of a matrix sits almost entirely in the numbers running down $\Sigma$ .

Rewrite it as a sum and the matrix becomes a stack of rank-1 layers:

A = \sum_{k} \sigma_k\, \boldsymbol{u}_k \boldsymbol{v}_k^{\top}

(4)

Each layer $\boldsymbol{u}_k \boldsymbol{v}_k^\top$ is the thinnest possible matrix — one direction of information — and $\sigma_k$ is how loudly that layer speaks. Keeping only the top $r$ layers is low-rank approximation, and the Eckart–Young theorem proves no better rank- $r$ truncation exists.

Read as a sum, it is the equation which says a matrix is a stack of thin sheets. The $\sigma_1$ sheet is the boldest and each one after it is fainter. Dropping everything past the top $r$ barely changes the picture — precisely because the sheets you dropped were faint to begin with.

LoRA — betting that the update is low rank

LoRA's equation has exactly the shape of that truncation:

W' = W + \Delta W = W + BA,\qquad B \in \mathbb{R}^{d\times r},\ A \in \mathbb{R}^{r\times d}

(5)

The change $\Delta W$ you want from fine-tuning is expressed not as a fat $d\times d$ matrix but as two thin factors $B\cdot A$ of rank $r$ . With $d=4096,\ r=8$ , the parameter count drops from $d^2 \approx 16.8$ M to $2dr \approx 66$ K — 0.4%.

Read in words, the equation says the original $W$ is never touched: you leave it frozen and bolt two thin factors on beside it. Gradients reach $B$ and $A$ and nothing else — which is also why one shared base model can have task-specific adapters swapped in and out.

It works because of one empirical bet: the change needed for fine-tuning concentrates in a few directions — it is effectively low rank. The figure below rebuilds a 28×28 matrix at rank $r$ . Slide $r$ up from 1 and watch the error collapse. 784 numbers reproduced by 56×r.

FIG 2Left, the original W; right, the rank-r reconstruction BA. Error collapses as r rises while the parameter count grows only as 56×r. LoRA trains only the right-hand side

To pin the correspondence down: LoRA's $B$ plays the role of the bundled $\sigma_k \boldsymbol{u}_k$ , and $A$ the bundled $\boldsymbol{v}_k^\top$ — except LoRA learns them by gradient descent instead of computing them. More in the LoRA paper walkthrough and SVD and low rank.

The dot product — the definition of "similar"

On the RAG side the protagonist is the dot product, which has two faces:

\boldsymbol{a}\cdot\boldsymbol{b} = \sum_i a_i b_i = \|\boldsymbol{a}\|\,\|\boldsymbol{b}\|\cos\theta

(6)

The left face is computational — multiply components, add them up. The right face is geometric — length × length × agreement of direction. Same direction: large and positive. Orthogonal: zero. Opposite: negative. As a device that turns "similar" into a number, nothing simpler exists.

The equals sign in the middle is the part which says something worth pausing on: a mechanical multiply-and-add is already measuring how well two directions agree. You only ever evaluate the left face, yet the angle on the right comes along for free — which is why retrieval can be a single dot product.

FIG 3Rotate b and the dot product runs positive → zero → negative. cos θ is the agreement of direction — the actual substance of "similarity"

To cancel the influence of length, divide by it. That is cosine similarity:

\mathrm{cos\_sim}(\boldsymbol{a},\boldsymbol{b}) = \frac{\boldsymbol{a}\cdot\boldsymbol{b}}{\|\boldsymbol{a}\|\,\|\boldsymbol{b}\|}

(7)

The dot product with length divided out — pure direction, always between −1 and 1. Plainly: this is the formula that lets a long document and a short one compete on equal footing.

Vector search — what actually happens inside RAG

RAG retrieval embeds the question and every document as vectors, then pulls the top-k documents closest to the question. "Close" means one of the three metrics you just met: dot product, cosine, or Euclidean distance.

They look interchangeable. They are not: an unnormalised dot product has a failure mode. Embedding length tracks document length and token frequency, so "loud" documents barge into the top-k even when they point the wrong way. Drag the query $q$ below and switch metrics. Cluster C is deliberately long — watch it start winning unfairly the moment you pick the dot product.

FIG 4Drag q and watch the top-5 change. Switch to "dot product" and the long vectors of cluster C cut in despite pointing elsewhere; cosine picks by direction alone

Normalise every vector to unit length, though, and all three metrics return the same ranking — because for unit vectors $\|\boldsymbol{a}-\boldsymbol{b}\|^2 = 2 - 2\,\boldsymbol{a}\cdot\boldsymbol{b}$ , so nearness in distance and largeness in dot product carry identical information. That one line is the entire reason production RAG systems normalise their embeddings and then search by inner product. The full pipeline is in RAG from scratch and the embeddings themselves in Embeddings from scratch.

In practice — where this shows up on the job

ML engineers doing fine-tuning — LoRA's $r$ is literally the $r$ in the low-rank figure. Start around r=8, lora_alpha=16, raise $r$ for harder tasks, and stop just before the error curve's collapse point: if doubling $r$ doesn't move quality, the change you need really was low rank.

Search and RAG infrastructure — normalise embeddings before they enter the index (faiss.normalize_L2; vector_cosine_ops in pgvector). Metric mismatch is the classic silent failure: an index built on inner product with only queries normalised shows up as vague quality loss and costs a day to trace.

Pitfalls — numerical eigen/singular routines shuffle order and sign freely; when taking "top r", always sort by absolute value. And with skewed embedding distributions, cosine scores can pile up above 0.8 for everything — rank by similarity, don't threshold it.

The two questions design reviews ask — why does LoRA still work at such a small rank, and why normalise embeddings at all? Because the change fine-tuning needs concentrates in a few directions, and because on the unit sphere dot product, cosine and distance all return the same ranking. Both figures above are the answer.

Summary

A matrix deforms space; an eigenvector is a direction that survives with only a scale change ( $Av=\lambda v$ )
SVD generalises this: any matrix is a stack of rank-1 layers, and truncating at $r$ is optimal (Eckart–Young)
LoRA's $\Delta W = BA$ bets the fine-tuning change is low rank, shrinking parameters to $2dr/d^2$
The dot product defines "similar"; divide by lengths for cosine; production RAG normalises, then searches by inner product

Four equations in total — each one paired with a slider you just moved.