Singular Value Decomposition and Low-Rank Approximation — the Math Behind LoRA

Starting from the 'rotate, stretch, rotate' picture, this article builds Singular Value Decomposition (SVD) from zero: matrices as stacks of rank-1 layers, why real-world data needs only a few of them, and how that single fact lets LoRA fine-tune a giant model with 0.4% of the parameters.

LoRA: Low-Rank Adaptation of Large Language Models

Giant matrices are smaller than they look

LoRA, now the default recipe for fine-tuning LLMs, adjusts models with billions of parameters using less than 1% as many trainable weights. On its face that sounds absurd — like claiming you can retune a machine with four billion dials by turning only a few million small ones.

Why does it work? The root of the answer is the star of this article: Singular Value Decomposition (SVD).

A matrix looks like a plain table of numbers, but matrices born from real data almost never carry as much information as their size suggests. A 1000×1000 matrix holds a million entries, yet in practice it often contains only "ten directions' worth" of information. SVD is the tool that measures this effective information content precisely and reorders the matrix's contents from most to least important. Once things are sorted, you can throw away the tail. What's left after the cut is a low-rank approximation — and applying that idea to fine-tuning gives you LoRA.

The metaphor: every transformation is "rotate, stretch, rotate"

Multiplying a vector by a matrix can be drawn as a deformation of space. Picture a thin rubber sheet with a circle drawn on it; apply a matrix, and the sheet warps, turning the circle into some other shape. The warps look endlessly varied — some shear diagonally, some flatten the sheet, some flip it over. Yet SVD makes a startlingly simple claim:

Any matrix's deformation can be reproduced exactly by three steps: ① rotate the sheet, ② stretch or shrink it along the axes, each by its own factor, ③ rotate again.

There are no exceptions. The matrix doesn't have to be square or symmetric — any matrix at all. Each step has a name: the first rotation is $V^\top$, the per-axis stretch factors are the singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$, and the final rotation is $U$.

Since rotations don't change shape, a matrix's entire "personality" is concentrated in the middle step. The circle becomes an ellipse, whose long axis corresponds to $\sigma_1$ (the most-stretched direction) and short axis to $\sigma_2$. An axis whose factor is nearly zero gets squashed flat — a direction along which the matrix transmits almost no information. The story of low-rank approximation has already begun right here.

The intuition: a matrix is a stack of layers, not a monolith

SVD supports a second reading, closer to this article's destination: it decomposes a matrix $A$ into a sum of rank-1 layers — the simplest possible matrices, each carrying information about just one direction.

$$A = \sigma_1 u_1 v_1^\top + \sigma_2 u_2 v_2^\top + \sigma_3 u_3 v_3^\top + \cdots$$

Read that as a shopping list rather than a formula: it is one line which says $A$ equals layer one plus layer two plus layer three and onward, each layer scaled by its own number $\sigma_i$ — and those numbers shrink as you move right.

In words: each layer $u_i v_i^\top$ is a single-purpose component that "measures how much of the input points along $v_i$, and outputs that amount in the direction $u_i$." The singular value $\sigma_i$ is that layer's volume knob — the bigger it is, the more the layer contributes to the matrix's overall behavior. Stack the loudest layers first, and you approach the original matrix quickly with very few of them.

"Measuring the component along $v_i$" is exactly the dot product — one number that says how much of one vector you can lay along another. Rotate the two vectors in the figure below to get that feeling into your hands.

So SVD is two things at once: a geometric decomposition into rotate–stretch–rotate, and an inventory audit that splits a matrix into single-purpose parts sorted by importance. From here on, we'll see just how powerful that inventory is, with a theorem and hard numbers.

The machinery: SVD as an equation

Everything so far compresses into one line.

Put in words, that line says: applying $A$ to a vector is the same as turning it ($V^\top$), stretching each axis by its own factor ($\Sigma$), and then turning it once more ($U$) — the rubber sheet from the last section, written down in symbols.

Here's what each symbol means. $A$ is any $m \times n$ matrix — the thing being decomposed. $U$ is an $m \times m$ orthogonal matrix whose columns $u_1, u_2, \ldots$ (the left singular vectors) are mutually perpendicular unit vectors, i.e. a rotation (or reflection). $V$ is likewise an $n \times n$ orthogonal matrix whose columns are the right singular vectors $v_1, v_2, \ldots$. And $\Sigma$ is an $m \times n$ matrix with the singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$ lined up on its diagonal, largest first, and zeros everywhere else.