LoRA vs. Full Fine-Tuning — What You Give Up, What You Get

LoRA squeezes a model's weight update into two thin matrices, ΔW = BA. This article starts from zero: why that is enough, what you actually trade away against full fine-tuning, how to pick the rank, and where merging quietly breaks.

LoRA: Low-Rank Adaptation of Large Language Models

Two ways to renovate

Picture renovating a finished building. You have two options. You can strip it back so every wall and column is accessible and rebuild anything you like. Or you can leave the structure untouched and bolt thin panels onto the few surfaces that need changing.

The first is full fine-tuning. The second is LoRA (Low-Rank Adaptation). The first gives you total freedom, at the cost of scaffolding and time. The second is a light job: you can peel the panels off to get the original back, or swap them for different ones. What you cannot do is change the structure itself.

This article walks through where that limitation actually shows up.

What full fine-tuning is really paying for

Start by breaking down why full fine-tuning is heavy. The surprise is that the weights themselves are not the expensive part — everything around them is.

Count what you need per parameter to train a 7-billion-parameter model with Adam:

• the weight itself (bf16) — 2 bytes
• its gradient (bf16) — 2 bytes
• Adam's first moment (fp32) — 4 bytes
• Adam's second moment (fp32) — 4 bytes
• an fp32 master copy of the weight for numerical stability — 4 bytes

That is roughly 16 bytes per parameter, or somewhere around 110 GB at 7B. Inference needs only the 2-byte weights — about 14 GB — so the moment you decide to train, the requirement inflates nearly eightfold. The gradients and optimizer state are the culprits. And this is before counting activations.

Here is the key observation: that inflation scales with the number of parameters you update, not the number you have. Shrink the set you update and the whole thing collapses. That is exactly what LoRA targets.

The low-rank assumption: ΔW = BA

Say fine-tuning turns some weight matrix $W_0$ into $W_0 + \Delta W$. LoRA's claim is that this $\Delta W$ can be approximated well enough by the product of two thin matrices.

In words, instead of storing a full $d \times k$ table of changes, the change is squeezed through $r$ narrow channels and then expanded back out to full size. $A$ projects the input down onto $r$ directions, and $B$ fans those $r$ directions back out to the output.

In plain terms: instead of storing a full $d \times k$ table of changes, you funnel the change through $r$ narrow channels and expand it back out. That $r$ is the rank — the number of channels. $A$ projects the input down onto $r$ directions; $B$ fans those $r$ directions back out to the output.

The parameter count makes the point. Updating a $4096 \times 4096$ matrix in full means about 16.8 million values. Replacing that with a LoRA at $r = 8$ costs $8 \times (4096 + 4096) = 65{,}536$ values — roughly 0.4% of the original. The gradients and optimizer state you have to hold during training shrink to that same 0.4%.

The forward pass looks like this:

In words, the output is what the original model already computed with the frozen $W_0$, plus a small bolted-on correction that is the only part that learns. $\alpha$ sets how strongly the correction is applied, and dividing by $r$ keeps its effective size from shifting when $r$ changes.

$x$ is the input vector and $h$ the output. The first term is what the original model already computed, and $W_0$ stays frozen — it never changes. The second term is the bolted-on correction, and it is the only thing that learns. $\alpha$ sets how strongly that correction is applied; dividing by $r$ normalizes it so that changing $r$ does not silently change the effective size of the update.

Why low rank is enough

"Can a correction really fit into two thin matrices?" — this is the crux of LoRA.

The justification is empirical. Adapting a pretrained model to a downstream task is less like installing new knowledge from scratch and more like reweighting abilities the model already has. Aghajanyan and colleagues showed experimentally that the effective number of degrees of freedom needed for fine-tuning is far smaller than the parameter count. LoRA takes that observation and expresses it in the convenient language of matrix rank.

You can feel the underlying behavior of low-rank approximation directly below. Add one rank at a time and watch how error trades against parameter count.

What you notice when you play with it: the first handful of ranks dominate, and each additional one buys rapidly less. That happens because the matrix's information is concentrated in a few strong directions. As long as that concentration exists, low-rank approximation is a good deal. When the change you need is spread evenly across every direction, the assumption breaks — and that is precisely where LoRA struggles, as we'll see.

One caveat about the widget: it approximates a matrix you already have, SVD-style. LoRA does something harder — it learns $A$ and $B$ by gradient descent without ever knowing the optimal $\Delta W$. The ceiling is the same; reaching it is a separate question. For the matrix theory itself, see SVD and low-rank approximation.

Why B starts at zero

There is one implementation detail that breaks training if you get it wrong: initialize $A$ with small random values and $B$ with exact zeros.

If $B = 0$ then $BA = 0$, so at step zero the output $h$ is bit-for-bit identical to the original model. Attaching a LoRA changes nothing until training starts nudging the correction into existence.

Initialize both randomly and you inject noise into the outputs before any learning happens, wrecking the pretrained behavior in the first few steps. Starting continuously from the original state is a design principle shared by essentially every bolt-on adaptation method.

Choosing the rank

The first question people ask in practice is what to set $r$ to. There is no principled optimum — it depends on how large a change the task demands. As a rough guide:

• Matching a tone, style, or output format — a small $r$ (single digits to low teens) is often plenty. You are changing how existing ability is expressed.
• Teaching new vocabulary or notation (internal jargon, a custom DSL, a rare language) — you need a larger $r$. This adds content rather than reweighting it.
• Injecting substantial domain knowledge — consider continued pretraining or full fine-tuning instead. Forcing this through LoRA is often the slower path.

For $\alpha$, common practice puts it at roughly one to two times $r$. Since $\alpha/r$ determines the effective strength of the update, doubling $r$ while leaving $\alpha$ alone halves how much the adapter actually does. A surprising share of "I raised the rank and it got worse" reports trace back to exactly this.

Higher rank is also not monotonically better. Add degrees of freedom on top of a small dataset and you get the familiar picture: training loss keeps falling while evaluation quietly degrades.

Which layers you attach to matters roughly as much as the rank. The original paper applied LoRA only to the attention projections $W_q$ and $W_v$, but later practice widely reports that covering all linear layers, including the MLP, works better — a point the QLoRA paper also makes. Before reaching for a higher $r$, try widening the target set first.

What you give up

Enough good news. Here is the honest cost.

1. There is a ceiling on expressiveness. By construction, $\Delta W$ has rank at most $r$. If the update you genuinely need is high-rank, no amount of training gets you there. Multiple reports find LoRA falling short of full fine-tuning on continued-pretraining-style workloads that pour in large volumes of new knowledge — often summarized as "learns less, forgets less."

2. Resistance to forgetting is also resistance to change. Because $W_0$ is frozen, the model's general capabilities are hard to damage (see catastrophic forgetting). But the same property preserves the base model's bad habits. If your goal is to remove an undesirable tendency baked in during pretraining, LoRA is the wrong tool.

3. Inference overhead is zero — if you merge. One of LoRA's real strengths is that $W_0 + \frac{\alpha}{r}BA$ can be computed ahead of time and folded into a single weight matrix. After merging, the model is structurally identical to any other, with no added latency. But if you keep adapters unmerged so you can hot-swap them, you pay for two matrix multiplications instead of one. On a single GPU serving per-customer adapters, that difference is real.

Where merging breaks

Merging is convenient, and less safe than people assume.

Adding two LoRAs does not add their capabilities. You can certainly compute $\Delta W_1 + \Delta W_2$, but each was learned as a correction around $W_0$, so their sum is a state neither one ever saw during training. Typically one dominates, or both degrade. Weighted averaging and orthogonalization schemes exist, but drop the naive expectation that summation composes.

Merging into a quantized base is not reversible. If you trained a LoRA on top of a 4-bit $W_0$ à la QLoRA, the result depends on the precision you merge into. Dequantize to fp16 and merge, and the artifact balloons in size; fold into 4-bit and you take on quantization error. Training at one numeric precision and shipping at another is a classic reason evaluation scores fail to reproduce. For the mechanics, see quantization from scratch.

The core, in code

A minimal PyTorch implementation is startlingly short.

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r: int, alpha: int):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False              # W0 stays frozen
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))  # ΔW = 0 at step 0
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)

Only A and B train, and only those two go to the optimizer. In practice you would not write this yourself — you would use Hugging Face peft and configure LoraConfig(r=..., lora_alpha=..., target_modules=[...], lora_dropout=...), then merge with model.merge_and_unload().

How this shows up on the job

Who reaches for it, and when. Anyone adapting an LLM to in-house data — ML engineers, application-side data scientists — should treat LoRA as the first thing to try. The cost-effective ordering is: (1) confirm that prompting and RAG genuinely aren't enough, (2) try LoRA, (3) reserve full fine-tuning for cases that still fall short and require large volumes of new knowledge. Jumping straight to full fine-tuning is a poor use of a GPU budget.

What you actually touch. In peft: r, lora_alpha, target_modules, lora_dropout. For quantized training, the bitsandbytes 4-bit settings. In the training loop, gradient_checkpointing and effective batch size. A practical starting point is to widen target_modules to every linear layer and begin around r=16, lora_alpha=32, adjusting from there.

Pitfalls that turn into incidents.

• Changing r without changing lora_alpha. The effective update strength shifts underneath you, so your "comparison" isn't comparing anything. Either hold $\alpha/r$ fixed or record explicitly that you changed it.
• Evaluating before merging, shipping after. With quantization in the mix these are different models. Evaluate the exact artifact you ship.
• Forgetting to load (or unload) the adapter. A model with no adapter attached does not error — it silently behaves like the base model. Put a one-case smoke test for the fine-tuned behavior into your deploy steps.
• Reusing the full-fine-tuning learning rate. LoRA updates far fewer parameters and grows a correction from zero, so it typically wants a learning rate an order of magnitude larger. Carry over the full-fine-tuning setting and it will look like nothing is happening at all.

A question worth being ready for. "Can you combine two separately trained LoRAs into one?" Answering "the weights add, but the capabilities don't — first consider mixing both datasets and training a single adapter" demonstrates you understand what the method actually is.

Summary

• LoRA constrains the update to the low-rank form $\Delta W = BA$, cutting trainable parameters to a few percent or less
• The savings come not from the weights but from gradients and optimizer state, which are only needed for what you update
• What you lose is a ceiling on expressiveness; for injecting large amounts of new knowledge, full fine-tuning can still win
• Behavior is governed by $\alpha/r$ and the target layers, not $r$ alone. Merging is convenient but breaks under multi-adapter composition and quantization

For how the original paper justified this design experimentally, see the LoRA paper walkthrough.