Paper Deep Dive — LoRA: Low-Rank Adaptation of Large Language Models: Why Low Rank Is Enough

A re-reading of the LoRA paper (Hu et al., 2021) grounded strictly in its own text: what BA actually means, the 21x amplification factor the authors measured, why r=1 sufficed, and the questions they left open.

LoRA: Low-Rank Adaptation of Large Language Models

Can you keep a hundred copies of a 175B model?

You want one large pre-trained model adapted to many downstream tasks. The obvious route is full fine-tuning, and its central defect is that the new model has as many parameters as the original (§1). What was a mere inconvenience for GPT-2 or RoBERTa large becomes a critical deployment problem for GPT-3 at 175 billion parameters. That is where the paper starts.

Its numbers are concrete. Full fine-tuning GPT-3 175B consumes 1.2TB of VRAM during training. With $r=4$ and only the query and value projection matrices adapted, the checkpoint you have to store drops from 350GB to 35MB — roughly a 10,000x reduction (§4.2). Footnote 4 is refreshingly blunt about what this does and doesn't buy you: you still need the 350GB base model at deployment, but hosting 100 adapted models costs $350\mathrm{GB}+35\mathrm{MB}\times100\approx354\mathrm{GB}$ rather than $100\times350\mathrm{GB}\approx35\mathrm{TB}$.

As an analogy, full fine-tuning rewrites the whole dictionary; LoRA seals the original and writes only the diff on a separate sheet. But that analogy only explains the storage bill. Why the diff can be so thin is the paper's real subject.

The intuition: adaptation amplifies directions that are already there

The hypothesis runs like this. Prior work (Li et al. 2018a; Aghajanyan et al. 2020) showed that over-parametrized models actually reside on a low intrinsic dimension. From that, the authors hypothesize that the change in weights during adaptation, $\Delta W$, also has a low "intrinsic rank" (§1).

Section 7.3 then measures it. For $W_q$ in the 48th layer of GPT-3, Table 7 compares Frobenius norms after projecting $W$ onto the subspace spanned by the singular vectors of $\Delta W$. At $r=4$, $\|U^\top W_q V^\top\|_F = 0.32$ while $\|\Delta W_q\|_F = 6.91$ — an amplification factor of roughly 21. Projecting onto $W_q$'s own top directions instead gives 21.67 (with $\|W_q\|_F = 61.95$), so $\Delta W$ is not simply repeating the dominant directions of $W$. A random matrix projection gives 0.02, so it isn't uncorrelated either. (For singular vectors, projections and low-rank approximation themselves, see Singular Value Decomposition and Low-Rank Approximation.)

The paper's conclusion: $\Delta W$ "amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model" (§7.3). Fine-tuning, on this reading, isn't building capability from nothing — it's pushing up a handful of equalizer sliders. Which is exactly why you don't need many sliders.

What was wrong with the existing options

Parameter-efficient adaptation predates LoRA. Section 3 sorts it into two families: adding adapter layers, and optimizing input-layer activations (prefix/prompt methods).

Adapters add inference latency. Adapter layers sit in sequence with the base model, so their compute is always on the critical path, and while you can prune layers, "there is no direct way to bypass the extra compute in adapter layers" (§3). Table 1 reports single forward passes on GPT-2 medium, averaged over 100 trials on an NVIDIA Quadro RTX8000. At batch 32 / sequence 512 the penalty is a modest +2.2% to +3.0%. At batch 1 / sequence 128, 19.8ms becomes 23.9ms (+20.7%) and 25.8ms (+30.3%). In online inference with small batches this is not negligible (§3, Appendix B). Sharding makes it worse, since the added depth needs more synchronous GPU operations like AllReduce and Broadcast.

Prefix methods are hard to optimize and eat your context. The paper observes that prefix tuning "is difficult to optimize and that its performance changes non-monotonically in trainable parameters," and adds the more fundamental objection: reserving part of the sequence length for adaptation necessarily reduces the sequence length available to the downstream task (§3).

LoRA's core claim — the update is low rank — is faster to feel than to read. Slide the rank and find where the error collapses.

The full mathematical floor — eigenvalues, SVD, dot products — is laid out in The linear algebra under LoRA and RAG.

The mechanism: freeze $W_0$, train only $BA$

For a pre-trained weight matrix $W_0\in\mathbb{R}^{d\times k}$, the update is constrained to a low-rank decomposition (§4.1):

$$W_0+\Delta W=W_0+BA,\quad B\in\mathbb{R}^{d\times r},\ A\in\mathbb{R}^{r\times k},\ r\ll\min(d,k)$$

Spelled out in words: whatever fine-tuning would have written into the weight matrix, LoRA forces you to write as one tall thin matrix times one short wide one. $A$ squeezes the input down to $r$ numbers and $B$ opens it back out, so every change has to pass through a waist $r$ values wide — and $r$ is deliberately chosen far smaller than either side of $W_0$.

and the forward pass is Equation (3):

$$h=W_0x+\Delta Wx=W_0x+BAx$$

The same line in words: the layer's output is what the frozen original would have produced, plus a correction computed from the very same input $x$. The two paths sit side by side rather than one after the other — and that is precisely what later lets you fold $BA$ back into $W_0$ and pay nothing for it at inference time.