LoRA from Scratch — Fine-Tuning a Giant by Training Two Thin Matrices

Full fine-tuning of a 7B model needs over 100 GB of optimizer state for a job whose actual content is small. LoRA's claim is that the *change* a model needs is low-rank, so it can be stored as a product of two thin matrices, ΔW = BA. This article builds that from the cost problem, takes the equations apart one line at a time, explains why B starts at zero and what α/r is for, counts the parameters for a real model, and ends on the three decisions that actually matter: the rank, the target layers, and what merging does at inference time.

LoRA: Low-Rank Adaptation of Large Language Models

The cost problem that LoRA exists to solve

You have a 7-billion-parameter language model and a few thousand examples of the tone you want it to write in. The job is small. The bill is not.

Full fine-tuning means updating every one of those 7 billion numbers, and for each one the optimizer has to keep company. With Adam in mixed precision the standard tally per parameter is roughly 4 bytes of master weights, 4 bytes of gradient, and 8 bytes of Adam's two moment estimates — about 16 bytes per parameter, which for 6.74 billion parameters is on the order of 108 GB before you have stored a single activation. That does not fit on one 80 GB accelerator. Then, having paid for it, you get a full copy of the model as output: another 14 GB on disk per task, per customer, per experiment.

Something is obviously out of proportion. Teaching a model a tone, a format, or a domain vocabulary is not remotely as much information as the model already contains, yet full fine-tuning charges you as if it were. LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2021, is the observation that you can pay in proportion to the change rather than in proportion to the model — and a concrete parameterisation that does it.

This article builds it from the ground: what "low rank" means and why anyone believed the update would be, the two-matrix substitution taken apart line by line, why one of the matrices starts at exactly zero, what the mysterious lora_alpha divides, the parameter count for a real model, and then the three knobs you will actually turn.

Prerequisite: rank, and what a low-rank matrix is missing

A quick definition, because everything below rests on it. The rank of a matrix is the number of linearly independent directions in it — the number of genuinely distinct rows (equivalently, columns), once you discard any that are just combinations of the others. A $4096 \times 4096$ matrix can have rank up to 4096; it is low-rank if its rank is far below that.

The useful consequence is a factorisation. Any matrix $M \in \mathbb{R}^{d \times k}$ of rank $r$ can be written as a product of a tall thin matrix and a short wide one:

$$M = \underbrace{B}_{d \times r} \cdot \underbrace{A}_{r \times k}$$

In words: any rank-$r$ matrix can be rewritten, exactly, as the product of two thin matrices.

Count what that saves. Storing $M$ directly takes $d \times k$ numbers. Storing the pair takes $d r + r k = r(d + k)$. For $d = k = 4096$ and $r = 8$: 16,777,216 numbers become 65,536 — a factor of 256. And the arithmetic runs the other way too: whatever $B$ and $A$ contain, their product $BA$ cannot have rank above $r$, because the information has to squeeze through an $r$-dimensional bottleneck on the way. That is the constraint LoRA imposes, and the whole question is whether it is a constraint the task can live with.

The intuition: the update is low-rank even though the model is not

Here is the leap. Nobody claims a pretrained weight matrix $W$ is low-rank — it is dense and uses its full capacity. LoRA's claim is about the difference between the fine-tuned weights and the pretrained ones:

$$\Delta W = W_{\text{fine-tuned}} - W_{\text{pretrained}}$$

In words: $\Delta W$ is simply how far fine-tuning moved the weights.

The bet is that this is low-rank, or close enough. Why would it be? Because pretraining has already installed the general machinery — grammar, world knowledge, the ability to follow a pattern. Adapting to a downstream task does not rebuild that machinery; it reweights and steers it. Steering is a much lower-dimensional act than building.

That intuition has an empirical antecedent. Aghajanyan et al. (arXiv:2012.13255) measured the intrinsic dimensionality of fine-tuning: they constrained updates to a randomly chosen low-dimensional subspace and found that surprisingly few dimensions were enough to reach most of full fine-tuning's performance — and that larger pretrained models had lower intrinsic dimension. If a task's update genuinely lives in a small subspace, then forcing $\Delta W$ to be low-rank is not throwing away something you needed; it is declining to pay for capacity the task never uses.

So the design follows: do not train $W$. Train a low-rank $\Delta W$ and add it.

The method, one line at a time

Line 1: the update is added, not overwritten

In words: the fine-tuned weights are the original weights plus a correction, and the original weights are left alone.

$W$ is the pretrained weight matrix, shape $d \times k$. $W'$ is the weight the fine-tuned model uses. $\Delta W$ is the correction. Nothing has been gained yet — this is just naming the difference — but it sets up the key move: $W$ is never modified. It stays exactly as pretraining left it, and all learning happens inside $\Delta W$.

Line 2: the correction is forced to be low-rank

In words: the correction can only be built by passing through a narrow channel of width $r$.

This is the whole idea in one line. Read it right to left, the way the data flows: $A$ takes a $k$-dimensional input and compresses it to $r$ dimensions; $B$ takes those $r$ numbers and expands them back out to $d$. The number $r$ is the rank — the width of the bottleneck — and it is the one hyperparameter that defines LoRA's capacity. Because everything must pass through $r$ dimensions, $\operatorname{rank}(BA) \le r$ is guaranteed by construction rather than by a penalty term.

Line 3: what actually runs in the forward pass

You never form $\Delta W$ as a matrix. For an input $x$, expand the product:

In words: the layer's original output, plus the output of a small side path.

Two paths, summed. $Wx$ is the original layer, untouched and computed exactly as before. $BAx$ is the LoRA branch, evaluated as $B(Ax)$ — first the cheap compression to $r$ numbers, then the expansion. The cost of the branch is $r(d+k)$ multiply-adds against the original layer's $dk$, so at $r = 8$ and $d = k = 4096$ the extra work is well under 1 %.

Line 4: freeze the base, train only the pair

In words: to lower the loss you may move only $A$ and $B$; $W$ is treated as a constant.

This is the line that produces every saving in the article. $W$ receives no gradient and no optimizer state; it is a constant. Gradients flow only into $A$ and $B$, so the optimizer's Adam moments — the 8 bytes per parameter that dominated the bill — are allocated for $r(d+k)$ numbers per layer instead of $dk$.

Note what does not get cheaper: the forward pass still runs through the full model, and activations must still be stored for backpropagation. LoRA cuts optimizer and gradient memory, not the base model's weight memory (you still load all 14 GB in bf16) and not the activation memory. This is the single most common misreading of what LoRA buys. The LoRA paper reports for GPT-3 175B a 10,000× reduction in trainable parameters and about a 3× reduction in GPU memory — the gap between those two numbers — more than three orders of magnitude — is exactly this distinction.

Line 5: initialisation — why $B$ starts at zero

$A$ is initialised randomly (Gaussian, or Kaiming-uniform in the reference implementation) and $B$ is initialised to all zeros. Therefore at step 0:

$$\Delta W = B A = 0 \cdot A = 0 \quad \Longrightarrow \quad W' = W$$

In words: at the moment training starts, the model produces exactly the same outputs as it would without LoRA.

The fine-tuned model begins as an exact copy of the pretrained one. No random noise is injected into a carefully trained network, so there is no initial loss spike and no need to warm back up to the base model's quality — training starts from the best available point and goes uphill from there.

Why zero only one of them, and not both? If $A$ and $B$ were both zero, the gradient of the loss with respect to each would be zero as well (each one's gradient is proportional to the other), and the pair would be stuck at zero forever — a dead symmetry. Zeroing exactly one breaks it: $B = 0$ makes the output zero, while $A \ne 0$ keeps $B$'s gradient alive, so the first step moves $B$ off zero and the branch comes to life.

Line 6: the scaling factor $\alpha/r$

In practice the branch is scaled:

In words: $\alpha$ sets how strong the correction is, and dividing by $r$ keeps that strength from drifting when you change the rank.

$\alpha$ (lora_alpha in every library) is a constant you set; it is divided by the rank. The reason is that $BA$'s typical magnitude grows with $r$ — a wider bottleneck sums more terms — so without the division, doubling the rank would also roughly double the effective size of the update, and your carefully tuned learning rate would no longer apply. Dividing by $r$ makes the learning rate approximately transferable across ranks, which is what turns $r$ into a capacity dial rather than a capacity-and-step-size dial.

The practical consequence is that only the ratio $\alpha/r$ matters for update strength. r=8, alpha=16 and r=16, alpha=32 have the same scale of 2 but different capacity. A frequent accident: someone raises $r$ from 8 to 64 to give the adapter more room, leaves $\alpha$ at 16, and silently shrinks the effective update by 8×, then concludes that higher rank "did nothing."

Counting it out on a real model

Take a 7B Llama-style model: hidden size $d = 4096$, 32 transformer layers. Apply LoRA at $r = 8$ to the query and value projections, the paper's recommended pair. Each of those is $4096 \times 4096$.

Per adapted matrix:

$$\underbrace{4096 \times 8}_{B} + \underbrace{8 \times 4096}_{A} = 65{,}536 \text{ parameters}$$

In words: each 4096×4096 matrix needs only about 65 thousand trainable numbers.

Two matrices per layer, 32 layers:

$$65{,}536 \times 2 \times 32 = 4{,}194{,}304 \approx 4.2\text{M trainable parameters}$$

In words: across the whole model, training touches only about 4.2 million numbers.

Against 6.74 billion total, that is 0.062 % — roughly one parameter in 1,600. And the consequences compound:

The last row is the honest part: LoRA does not shrink the model. What it eliminates is the optimizer state, and what it changes operationally is the last-but-one row — a 16 MB adapter instead of a 14 GB checkpoint. That is why a single served base model can host hundreds of task-specific adapters, swapped per request, which is the deployment pattern LoRA really unlocked.

In code the substitution is unremarkable, which is part of the appeal:

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=8, alpha=16):
        super().__init__()
        self.base = base
        for p in self.base.parameters():   # freeze the pretrained weights
            p.requires_grad = False
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))  # zero => ΔW = 0
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T) @ self.B.T

Three details carry the whole method: requires_grad = False on the base, zeros for B, and alpha / r as the scale.

Three pitfalls, which are also the only three decisions

Choosing the rank

$r$ is a capacity ceiling, and both directions have a failure mode. Too small and the update physically cannot express what the task needs — the model underfits no matter how long you train, because you have constrained the solution to a subspace that does not contain a good answer. Too large and you lose LoRA's advantages (more parameters, bigger adapter, more room to overfit a small dataset) without necessarily gaining quality.

The empirical finding that surprises people is how small "enough" usually is. The LoRA paper reports that $r = 1$ was already competitive on their tasks, and that the top singular directions of adapters trained at $r=8$ and $r=64$ overlapped heavily — the extra rank was largely spent on directions that contributed little. The rule of thumb that follows: start at $r = 8$ or $16$ for style, format, and tone; consider $32$–$64$ when teaching genuinely new domain content. And since raising $r$ without raising $\alpha$ shrinks the update, change them together or hold the ratio fixed on purpose.

If you see underfitting at low rank, do not raise $r$ first — check the learning rate. LoRA typically wants a learning rate an order of magnitude higher than full fine-tuning (say $1\text{e-}4$ rather than $1\text{e-}5$), because the parameters being trained are few, freshly initialised, and scaled down by $\alpha/r$. A great many "LoRA cannot learn this" reports are a full-fine-tuning learning rate left in the config.

Choosing which layers to adapt

LoRA is applied per matrix, so you decide which ones. The original paper, working under a fixed parameter budget, found that spreading a small budget over $W_q$ and $W_v$ beat concentrating it on one matrix at higher rank — coverage mattered more than depth. Current practice has moved further in that direction: targeting all the linear projections, attention ($q, k, v, o$) and the MLP ones, generally works better when budget is not the binding constraint, and this is what tools mean by target_modules="all-linear".

The trap is silent under-application. If target_modules names a pattern that matches nothing in your architecture — easy to do, since projection names differ between model families — you get a model that trains, logs a falling loss from the handful of matrices that did match, and learns far less than you intended. Print the trainable parameter count before every run (print_trainable_parameters() in PEFT). If it reads 0.01 % when you expected 0.5 %, your pattern missed.

A second trap concerns anything the task needs that no adapted matrix can reach. If you extend the tokenizer with new tokens, the embedding and output-head rows for those tokens are frozen and random, and no amount of attention-layer adaptation fixes them; those modules have to be trained (or the tokens avoided).

Merging at inference time

Because the branch is linear and added, it can be folded into the base weights once training is done:

$$W_{\text{merged}} = W + \frac{\alpha}{r} B A$$

In words: once training ends, the correction can be added into the original weights, leaving one ordinary matrix.

Do that and the LoRA matrices disappear. The served model has the original architecture, the original parameter count, and — this is LoRA's headline advantage over bottleneck adapter layers, which insert extra modules into the forward pass — exactly zero added inference latency.

Three things go wrong here in practice.

Merging into a quantized base loses precision. With QLoRA the base weights are stored in 4-bit while $A$ and $B$ train in higher precision. Adding a 16-bit update into a 4-bit tensor means re-quantising, and the small update can be partly rounded away, which shows up as a quality drop between your evaluation (adapter attached to a dequantised base) and production (merged). Either keep the adapter unmerged, or merge into the original full-precision weights rather than the quantized copy.

Merging forfeits the thing you wanted LoRA for. An unmerged adapter can be swapped per request, letting one loaded base model serve many tasks; a merged one is a whole new model. Merge when you serve one task at high volume, keep adapters separate when you serve many.

Two adapters do not simply add. $W + \Delta W_1 + \Delta W_2$ is well-defined arithmetic but not a reliable composition of two behaviours: the updates were each trained assuming the other was absent, and their directions can conflict. Concatenating two rank-8 adapters also gives something of rank up to 16, not rank 8. Treat multi-adapter merging as an experiment to evaluate, not an operation that works by construction.

How it is used in practice

The people who touch LoRA day to day are machine learning engineers adapting a published foundation model to their own tone, output format, or domain vocabulary. The typical trigger is one of two situations: prompt engineering alone will not make the output format stable, or there is no room to store and serve a full copy of the model per task. The same mechanism is also widely used with image-generation models to teach a style or a character.

The settings you actually touch. In Hugging Face PEFT, a handful of LoraConfig fields is the whole surface:

• r and lora_alpha — capacity and strength. Move them together with the ratio $\alpha/r$ in mind, as above
• target_modules — which linear layers to adapt. Names differ between model families, so check the real names with model.named_modules() before writing the pattern
• lora_dropout — dropout applied to the LoRA branch's input; a guard against overfitting when data is scarce
• modules_to_save — modules to train and save in full rather than through LoRA (for example the embeddings and output head after adding tokens)
• Check the fraction with print_trainable_parameters(), save with save_pretrained() (which writes only the adapter weights and adapter_config.json), and merge with merge_and_unload()

Pitfalls that cause incidents if you do not know them.

• An adapter only fits the base model it was trained on. It stores nothing but the difference $\Delta W$, so loading it onto a different base — including a different revision under the same name, or a different quantized build — will, if the shapes match, load without any error and simply produce wrong outputs. Keep base_model_name_or_path and the revision in adapter_config.json in sync with what you serve.
• Do not change the prompt format between training and inference. The adapter learns the training data's format along with its content (chat template, delimiters). Serve with a different format and the adapter looks like it is doing nothing.
• Evaluate the exact artifact you ship. As shown above, merged vs. unmerged and quantized vs. not can be different models.

Questions you get in interviews and design reviews.

• "Where does LoRA save memory, and where does it not?" → It saves gradients and optimizer state. Base weights and activations are not reduced
• "Why initialise $B$ to zero, and why not both?" → So training starts identical to the base model. With both at zero, each gradient is zero because of the other, and nothing moves
• "You raised r and nothing improved. What do you suspect?" → That $\alpha$ was left alone and the effective scale shrank, that the learning rate is still the full-fine-tuning one, or that target_modules is not actually matching
• "Should you merge?" → Merge for zero latency when serving one task at volume; keep adapters separate and swap them when serving many tasks

Summary

• Full fine-tuning charges ~16 bytes of optimizer state per parameter — over 100 GB for a 7B model — for a change that is far smaller than the model
• LoRA freezes $W$ and learns the update as a product of two thin matrices: $W' = W + \frac{\alpha}{r} BA$, with $r \ll \min(d,k)$ forcing the update to be low-rank by construction
• $B$ starts at zero so training begins as an exact copy of the pretrained model; $A$ is random so the symmetry breaks and gradients can flow
• $\alpha/r$ scaling makes the learning rate roughly transferable across ranks — only the ratio sets update strength, which is why raising $r$ alone can weaken the update
• At $r = 8$ on $q$ and $v$, a 7B model trains 4.2 M parameters (0.062 %) and ships a ~16 MB adapter instead of a 14 GB checkpoint — but the base weights and activations still occupy full memory
• The three real decisions: the rank, which matrices to target (verify the count, do not trust the pattern), and whether to merge — merging buys zero latency but gives up per-request adapter swapping and degrades on a quantized base

For the trade-off against full fine-tuning in practice, see LoRA vs. Full Fine-Tuning. For a close reading of the original paper's own measurements, see the LoRA paper deep dive. If the rank and factorisation language felt shaky, start with Linear Algebra for AI.

Exercises

1. A layer is $d \times k = 4096 \times 11008$ (a Llama MLP projection). How many trainable parameters does a LoRA adapter at $r = 16$ add, and what fraction is that of the original matrix?
2. You train with r=8, lora_alpha=16 and the results are good. A colleague copies your config, sets r=64 to "give it more capacity", changes nothing else, and reports that the model barely moved from the base. What happened, and what value of lora_alpha restores the original update strength?
3. Explain why initialising both $A$ and $B$ to zero would prevent learning entirely, while initialising $B$ alone to zero does not.
4. Your evaluation runs the adapter attached to a 4-bit quantized base and scores well. You merge the adapter for production and the score drops. Give the likely cause and two ways to avoid it.