LoRA Paper Explained: Low-Rank Adaptation of Large Language Models (Hu et al., 2021) — Why Low Rank Is Enough

The LoRA paper (Hu et al., 2021) explained, grounded strictly in its own text: what BA actually means, the 21x amplification factor the authors measured, why r=1 sufficed, and the questions they left open.

LoRA: Low-Rank Adaptation of Large Language Models

Can you keep a hundred copies of a 175B model?

You want one large pre-trained model adapted to many downstream tasks. The obvious route is full fine-tuning, and its central defect is that the new model has as many parameters as the original (§1). What was a mere inconvenience for GPT-2 or RoBERTa large becomes a critical deployment problem for GPT-3 at 175 billion parameters. That is where the paper starts.

Its numbers are concrete. Full fine-tuning GPT-3 175B consumes 1.2TB of VRAM during training. With $r=4$ and only the query and value projection matrices adapted, the checkpoint you have to store drops from 350GB to 35MB — roughly a 10,000x reduction (§4.2). Footnote 4 is refreshingly blunt about what this does and doesn't buy you: you still need the 350GB base model at deployment, but hosting 100 adapted models costs $350\mathrm{GB}+35\mathrm{MB}\times100\approx354\mathrm{GB}$ rather than $100\times350\mathrm{GB}\approx35\mathrm{TB}$.

As an analogy, full fine-tuning rewrites the whole dictionary; LoRA seals the original and writes only the diff on a separate sheet. But that analogy only explains the storage bill. Why the diff can be so thin is the paper's real subject.

The intuition: adaptation amplifies directions that are already there

The hypothesis runs like this. Prior work (Li et al. 2018a; Aghajanyan et al. 2020) showed that over-parametrized models actually reside on a low intrinsic dimension. From that, the authors hypothesize that the change in weights during adaptation, $\Delta W$, also has a low "intrinsic rank" (§1).

Section 7.3 then measures it. For $W_q$ in the 48th layer of GPT-3, Table 7 compares Frobenius norms after projecting $W$ onto the subspace spanned by the singular vectors of $\Delta W$. At $r=4$, $\|U^\top W_q V^\top\|_F = 0.32$ while $\|\Delta W_q\|_F = 6.91$ — an amplification factor of roughly 21. Projecting onto $W_q$'s own top directions instead gives 21.67 (with $\|W_q\|_F = 61.95$), so $\Delta W$ is not simply repeating the dominant directions of $W$. A random matrix projection gives 0.02, so it isn't uncorrelated either. (For singular vectors, projections and low-rank approximation themselves, see Singular Value Decomposition and Low-Rank Approximation.)

The paper's conclusion: $\Delta W$ "amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model" (§7.3). Fine-tuning, on this reading, isn't building capability from nothing — it's pushing up a handful of equalizer sliders. Which is exactly why you don't need many sliders.

What was wrong with the existing options

Parameter-efficient adaptation predates LoRA. Section 3 sorts it into two families: adding adapter layers, and optimizing input-layer activations (prefix/prompt methods).

Adapters add inference latency. Adapter layers sit in sequence with the base model, so their compute is always on the critical path, and while you can prune layers, "there is no direct way to bypass the extra compute in adapter layers" (§3). Table 1 reports single forward passes on GPT-2 medium, averaged over 100 trials on an NVIDIA Quadro RTX8000. At batch 32 / sequence 512 the penalty is a modest +2.2% to +3.0%. At batch 1 / sequence 128, 19.8ms becomes 23.9ms (+20.7%) and 25.8ms (+30.3%). In online inference with small batches this is not negligible (§3, Appendix B). Sharding makes it worse, since the added depth needs more synchronous GPU operations like AllReduce and Broadcast.

Prefix methods are hard to optimize and eat your context. The paper observes that prefix tuning "is difficult to optimize and that its performance changes non-monotonically in trainable parameters," and adds the more fundamental objection: reserving part of the sequence length for adaptation necessarily reduces the sequence length available to the downstream task (§3).

LoRA's core claim — the update is low rank — is faster to feel than to read. Slide the rank and find where the error collapses.

The full mathematical floor — eigenvalues, SVD, dot products — is laid out in The linear algebra under LoRA and RAG.

The mechanism: freeze $W_0$, train only $BA$

For a pre-trained weight matrix $W_0\in\mathbb{R}^{d\times k}$, the update is constrained to a low-rank decomposition (§4.1):

$$W_0+\Delta W=W_0+BA,\quad B\in\mathbb{R}^{d\times r},\ A\in\mathbb{R}^{r\times k},\ r\ll\min(d,k)$$

Spelled out in words: whatever fine-tuning would have written into the weight matrix, LoRA forces you to write as one tall thin matrix times one short wide one. $A$ squeezes the input down to $r$ numbers and $B$ opens it back out, so every change has to pass through a waist $r$ values wide — and $r$ is deliberately chosen far smaller than either side of $W_0$.

and the forward pass is Equation (3):

$$h=W_0x+\Delta Wx=W_0x+BAx$$

The same line in words: the layer's output is what the frozen original would have produced, plus a correction computed from the very same input $x$. The two paths sit side by side rather than one after the other — and that is precisely what later lets you fold $BA$ back into $W_0$ and pay nothing for it at inference time.

During training $W_0$ is frozen and receives no gradient updates. Three design points matter.

1. Initialization: $A$ is random Gaussian, $B$ is zero, so $\Delta W=BA$ is zero at the start of training and the model is exactly the base model (§4.1).
2. Scaling: $\Delta Wx$ is scaled by $\frac{\alpha}{r}$, with $\alpha$ constant in $r$. The paper notes that "when optimizing with Adam, tuning $\alpha$ is roughly the same as tuning the learning rate if we scale the initialization appropriately," and states plainly that they set $\alpha$ to the first $r$ they try and do not tune it. The point of the scaling is to avoid retuning hyperparameters as you vary $r$ (§4.1).
3. Merging: at deployment you explicitly compute and store $W=W_0+BA$ and run inference normally. Switching tasks means subtracting $BA$ and adding a different $B'A'$. This is what guarantees zero additional inference latency by construction (§4.1) — the clean separation from adapter layers.

On placement: of the four self-attention matrices ($W_q,W_k,W_v,W_o$) and the two MLP matrices, the paper restricts itself to attention weights and freezes the MLP, citing simplicity and parameter efficiency, and explicitly leaves adaptation of MLP layers, LayerNorms and biases to future work (§4.2). For what each of those four matrices actually does, see Attention from Scratch.

The authors also argue LoRA is a generalization of full fine-tuning: apply it to all weight matrices, train the biases too, and raise $r$ to the rank of the pre-trained matrices, and you roughly recover full fine-tuning's expressiveness. As trainable parameters grow, LoRA converges toward training the original model, whereas adapter methods converge to an MLP and prefix methods to a model that cannot take long inputs (§4.1).

What the experiments showed

Evaluation spans RoBERTa and DeBERTa (GLUE), GPT-2 (E2E NLG), and GPT-3 175B (WikiSQL, MNLI, SAMSum), all on NVIDIA Tesla V100 (§5). In most experiments LoRA is applied only to $W_q$ and $W_v$ (§5.1).

• GLUE (Table 2): RoBERTa base scores 86.4 average with 125.0M fine-tuned parameters, versus 87.2 for LoRA at 0.3M. RoBERTa large: 88.9 at 355.0M, 89.0 for LoRA at 0.8M. DeBERTa XXL: 91.1 at 1500.0M, 91.3 for LoRA at 4.7M.
• GPT-2 / E2E NLG (Table 3): GPT-2 M fine-tuned at 354.92M gives BLEU 68.2; LoRA at 0.35M gives 70.4. GPT-2 L: 68.5 at 774.03M versus 70.4 for LoRA at 0.77M.
• GPT-3 175B (Table 4): full fine-tuning at 175,255.8M parameters gives WikiSQL 73.8 / MNLI-m 89.5 / SAMSum 52.0-28.0-44.5. LoRA at 4.7M gives 73.4 / 91.7 / 53.8-29.8-45.9; LoRA at 37.7M gives 74.0 / 91.6. The paper states seed-driven fluctuation of ±0.5% on WikiSQL and ±0.1% on MNLI-m.
• Which matrices to adapt (§7.1, Table 5): at a fixed 18M-parameter budget (about 35MB in FP16), WikiSQL scores 70.4 for $W_q$ alone, 73.0 for $W_v$ alone, and 73.7 for $\{W_q,W_v\}$ at $r=4$. The conclusion: "even a rank of four captures enough information in $\Delta W$ such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank."
• The optimal $r$ (§7.2, Table 6): with $\{W_q,W_v\}$, $r=1$ already gives WikiSQL 73.4 / MNLI 91.3, essentially matching $r=64$ (73.5 / 91.4). With $W_q$ alone, $r=1$ falls to 68.8 — consistent with the §7.1 finding that breadth beats rank.
• Why raising $r$ adds little (§7.2, Figure 3): decomposing the learned $A$ matrices from $r=8$ and $r=64$ runs and measuring subspace overlap, the top singular directions overlap strongly while the rest do not. The shared subspace with normalized similarity above 0.5 has dimension 1, which the paper offers as the explanation for why $r=1$ works at all (analysis on layer 48 of 96).
• Low-data regime (Appendix F.3, Table 16): restricted to 100 MNLI examples, full fine-tuning scores 60.2 versus 63.8 for LoRA. PrefixEmbed manages 37.6, only slightly above the 33.3 random baseline.

Limitations and open questions

Acknowledged by the paper:

• Mixing tasks within a batch is awkward. If you absorb $A$ and $B$ into $W$ to eliminate latency, it is not straightforward to batch inputs for different tasks — each needing different $A$ and $B$ — in a single forward pass. The paper notes you can skip merging and select modules dynamically where latency isn't critical (§4.2).
• MLP, LayerNorm and bias adaptation are untested. Experiments cover self-attention weights only (§4.2).
• Small $r$ is not claimed to be universal. Footnote 6 offers a thought experiment: if the downstream task were in a different language than pre-training, retraining the whole model (LoRA with $r=d_{model}$) could certainly beat a small $r$. It is a thought experiment, not a measurement.
• Choosing which matrices to adapt is heuristic. The conclusion (§8) asks the question of itself: "We mostly depend on heuristics to select the weight matrices to apply LoRA to. Are there more principled ways to do it?" It also states that the mechanism behind fine-tuning and LoRA "is far from clear."
• The subspace analysis is thinly sampled. Figures 3 and 4 analyze layer 48 of 96; Appendix H.1 asserts the conclusion holds for other layers, but the evidence presented is a handful of visualizations.
• The best $r$ is model- and task-dependent. Appendix H.2 reports that on GPT-2 Medium with E2E, validation loss peaks at $r=16$ and BLEU at $r=4$ — so the GPT-3 result that $r=1$ suffices does not transfer unchanged.

Note as of this writing in 2026 (the author's annotation, not a claim from the paper): the experiments here target 2021-era models (RoBERTa, DeBERTa, GPT-2, GPT-3 175B). LoRA has since spawned many variants, including combinations with quantized base models, but every number in this article is from this paper; none come from later work.

Practical takeaways

The following are the author's operational readings of what the paper reports.

• Widen coverage before raising $r$. Tables 5 and 6 point the same way: a small $r$ across $\{W_q,W_v\}$ beats a large $r$ on $W_q$ alone.
• A bigger $r$ does not automatically help. In Table 6 there are combinations where $r=64$ ties or slightly trails $r=1$.
• Don't retune $\alpha$ every run. The paper's own practice is to set $\alpha$ to the first $r$ tried and leave it alone (§4.1). The $\alpha/r$ scaling exists precisely so you don't have to re-tune the learning rate when you sweep $r$ — moving both at once throws that away.
• Don't reuse the full fine-tuning learning rate. In the GPT-3 setup (Appendix D.4), full fine-tuning uses 5.00E-06 while LoRA uses 2.00E-04, nearly two orders of magnitude apart (AdamW, 2 epochs, batch 128, weight decay 0.1).
• State batch size and sequence length when comparing against adapters. Per Table 1, the latency gap is nearly invisible at batch 32 / sequence 512 and around 30% at batch 1 / sequence 128. "Adapters are slow" is a conditional claim.
• Keep the $B=0$ initialization. The property that the model starts identical to the base model comes from exactly that choice (§4.1).
• Stacking with prefix methods isn't uniformly good. Appendix E reports LoRA+PrefixEmbed beating both LoRA alone and prefix-embedding tuning on WikiSQL, failing to beat LoRA alone on MultiNLI, and LoRA+PrefixLayer coming out slightly worse than LoRA despite more trainable parameters.

How this shows up at work — LoRA in practice

The reason this 2021 paper still gets read on the job is that its footnote-4 arithmetic is a serving-architecture argument, not a training trick. Two roles hit it constantly.

The ML engineer who has to make one base model serve many behaviours. A request arrives to add a domain-specific behaviour — a support tone, a schema-constrained output, a customer's house style — and the honest full fine-tuning answer is another checkpoint the size of the original. LoRA turns that into a per-task artifact small enough to keep dozens of, alongside a single shared base. The decision you are actually making is where the tenancy boundary sits: one merged model per task, or one base with modules selected per request.

The person who owns the latency budget. Whether an adaptation method can ship behind an existing serving SLO is decided by §3 and §4.1, not by an accuracy table. The mergeability argument — $W=W_0+BA$ computed once, offline — is what lets you promise the serving path is bit-for-bit the shape it was before.

The knobs you touch in a LoRA config are the paper's symbols under slightly different names: rank r is $r$, lora_alpha is $\alpha$ from the $\alpha/r$ scaling, and target_modules is the §7.1 question of which matrices to attach to — the paper's own choice being the query and value projections. Serving then needs an explicit merge step that folds $BA$ into the base weights before the model is loaded for inference.

Places this goes wrong in production:

• Shipping unmerged. Selecting modules dynamically is legitimate and the paper says so (§4.2), but it puts the low-rank path back on the critical path — you are now paying the adapter cost the method existed to remove, and Table 1 says that cost is worst at batch 1 with short sequences, which is exactly where interactive serving lives.
• Sweeping $r$ and $\alpha$ together. The $\alpha/r$ scaling only earns its keep if $\alpha$ stays put while $r$ moves (§4.1). Tune both and you have re-tuned the learning rate without meaning to, then attributed the result to rank.
• Inheriting the full fine-tuning learning rate. Appendix D.4 puts the two nearly two orders of magnitude apart. A LoRA run at a full fine-tuning LR looks like the method failing rather than the config being wrong.
• Losing track of which base checkpoint the adapter was trained against. $BA$ is a delta and means nothing except relative to the $W_0$ it was fitted to. Version the base weights next to the adapter; a quiet base upgrade silently invalidates every adapter you have stored.
• Assuming merged adapters can be mixed inside one batch. They cannot, straightforwardly (§4.2). If the roadmap says "many tenants, one batch," that constraint belongs in the design review, not in the incident afterwards.
• Quoting a latency comparison with no batch size or sequence length attached. Per Table 1 the same comparison is either negligible or roughly 30%, depending entirely on numbers people leave out of slides.

The question worth being able to answer out loud: why does zero added latency hold for LoRA but not for adapters? The answer is not "LoRA is smaller." It is that $BA$ is applied in parallel to $W_0$ on the same input, so it can be summed into the weights ahead of time, while an adapter layer sits in sequence and there is no arithmetic that makes a serial block disappear.

Summary

• LoRA is nothing more than "add a low-rank delta $BA$ to a frozen $W_0$," and merging guarantees no added inference latency by construction (§4.1).
• The case for low rank has two layers: the hypothesis that adaptation has low intrinsic rank (§1), and the measurement that $\Delta W$ amplifies directions already present but unemphasized in $W$ by a factor of roughly 21 (§7.3, Table 7).
• The untested territory is equally explicit: MLP layers, multi-task batching, and a principled way to choose which matrices to adapt (§4.2, §8).

Original paper: Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (arXiv:2106.09685, 2021 / v2)