QLoRA — Fine-Tuning a 65B Model on One GPU

A 65-billion-parameter model in 16-bit weights is 130 GB before you have trained anything. QLoRA gets that onto a single 48 GB card by storing the frozen weights in a 4-bit format shaped like the weights themselves, quantizing the quantization constants, and paging the optimizer to CPU — while gradients still flow through those 4-bit weights into a small 16-bit adapter.

LoRA: Low-Rank Adaptation of Large Language Models

The memory wall, stated in numbers

Fine-tuning a large language model on your own data is, conceptually, the easy part: take the pretrained weights, run your examples through, adjust. The obstacle is arithmetic that has nothing to do with machine learning.

Take a 65-billion-parameter model. Weights alone, in the usual 16-bit BF16 format (two bytes per number), come to

$$65 \times 10^9 \text{ params} \times 2 \text{ bytes} = 130\ \text{GB}$$

In words: how many numbers you store, times how many bytes each one takes, is exactly how much memory you need.

That is the model sitting still, before a single gradient exists. Full fine-tuning needs three more things of comparable size: the gradients (another copy, ~130 GB), and the Adam optimizer's two running statistics per parameter, conventionally kept in 32-bit, which is $65 \times 10^9 \times 4 \times 2 = 520$ GB. Add activations and you are past 800 GB — a multi-node cluster, for one fine-tune.

The largest single accelerators available when QLoRA was published held 48 GB (an A6000) or 80 GB (an A100). The gap was not a factor of two. QLoRA (Dettmers et al., 2023) closes it, and the headline result is blunt: a 65B model fine-tuned on one 48 GB GPU in about a day, matching the quality of 16-bit fine-tuning. This article builds the method from its prerequisite — LoRA — through the three pieces that make 4 bits survivable, and ends with the thing people most often get wrong about it.

Prerequisite: LoRA's low-rank update

QLoRA is not a new training algorithm. It is LoRA with the frozen part compressed, so LoRA has to be clear first. (The full treatment is in LoRA vs Full Fine-Tuning; this is the one paragraph you need here.)

Fine-tuning changes a weight matrix $W$ into $W + \Delta W$. LoRA's bet is that the change $\Delta W$, unlike $W$ itself, is close to low-rank — that it can be written as the product of two thin matrices. So instead of storing and updating a full $d \times k$ matrix of changes, you store two:

$$\Delta W = BA, \qquad B \in \mathbb{R}^{d \times r},\quad A \in \mathbb{R}^{r \times k},\quad r \ll \min(d,k)$$

Read that as a sentence: squeeze the update through a bottleneck of width $r$ — $A$ projects the input down to $r$ dimensions, $B$ expands it back out — so the whole modification is described by $r(d+k)$ numbers instead of $dk$. The forward pass becomes

$$h = Wx + BAx$$

with $W$ frozen and only $A, B$ trainable. For a $4096 \times 4096$ matrix at $r = 16$, that is $16 \times 8192 = 131{,}072$ trainable numbers in place of $16.7$ million — a factor of 128.

The consequence that matters for memory: gradients and optimizer states exist only for $A$ and $B$. The 520 GB of Adam state disappears, because Adam is now tracking a few hundred million adapter parameters instead of 65 billion. What LoRA does not do is shrink $W$. Those 130 GB of frozen weights still have to be resident on the GPU, because every forward pass multiplies by them.

That is exactly the hole QLoRA fills.

Where the memory actually goes

Write out the budget for a 65B LoRA fine-tune, and one line dwarfs the others:

LoRA already removed the big optimizer term. The remaining problem is a single number: 130 GB of weights that never change. And weights that never change are an unusually good target for compression, because you only ever need to read them. There is no requirement that they be stored in a format you can accurately accumulate small updates into — that requirement is what forces 16 or 32 bits during normal training. Frozen weights only have to be read back accurately enough to compute the same forward pass.

QLoRA's whole design follows from that observation: store $W$ in 4 bits, dequantize it on the fly to 16 bits for each matrix multiply, and keep the parts that actually get updated — the adapters — in full 16-bit precision.

At 4 bits, 65B parameters come to roughly

$$65 \times 10^9 \times 4\ \text{bits} = 2.6 \times 10^{11}\ \text{bits} = 32.5\ \text{GB}$$

In words: cut the bits per number from 16 to 4, and the space the weights occupy shrinks by the same factor of four.

Suddenly the model fits on the 48 GB card with room for adapters, activations, and a batch. The rest of the article is about making 4 bits accurate enough that the fine-tune still works.

Quantization in one paragraph

Quantization means replacing a continuous range of numbers with a small finite set of levels, and storing only the index of the nearest level. With 4 bits you get $2^4 = 16$ levels. The standard recipe is absmax block-wise quantization: chop the weight tensor into blocks (QLoRA uses blocks of 64 values), and for each block record one high-precision scaling constant

$$c = \max_i |x_i|$$

which says: remember one number per block — the largest magnitude in it — and use it as the ruler everything else in the block is measured against. You then store each value as the index of the level nearest to $x_i / c \in [-1, 1]$. Dequantizing is a lookup and a multiply: $\hat{x}_i = c \cdot q_{\text{index}}$.

Blocks matter for a reason worth naming. If you used a single scale for a whole tensor, one outlier weight — and large models are full of them — would stretch $c$ so far that every ordinary weight in the tensor collapses onto the same couple of levels. Small blocks confine the damage: an outlier ruins the resolution of its own 64 neighbours and nobody else's. (The general picture, including what changes for inference, is in Quantization Explained.)

The open question is which 16 levels. That is where QLoRA's first contribution comes in.

NF4: sixteen levels shaped like the weights

The obvious choice is 16 evenly spaced levels. It is also the wrong one, and the reason is an empirical fact about neural networks: trained weights within a layer are approximately normally distributed, zero-centred and bell-shaped. Evenly spaced levels spend the same precision on the tails, where almost no weights live, as on the centre, where almost all of them do.

The fix is quantile quantization: choose the levels so that each level is used by about the same number of weights. If the 16 bins are equiprobable, every bin carries the same $1/16$ of the distribution's mass, and the average error is as small as it can be for 16 symbols — this is the information-theoretically optimal choice for data from a known distribution. Doing this honestly would mean estimating the quantiles of every block's actual data, which is expensive and noisy.

QLoRA's shortcut is to assume the distribution instead of measuring it. Since a block of weights divided by its absmax is approximately a normal distribution squeezed into $[-1, 1]$, you can compute the quantiles of a standard normal once, offline, and use that fixed table forever. The resulting 4-bit data type is NF4 (4-bit NormalFloat): a fixed, non-uniform table of 16 values in $[-1, 1]$, with levels packed densely near zero and spread out near $\pm 1$.

Two details of the table are worth knowing because they explain small oddities in implementations. First, the levels are asymmetric — 8 negative, 8 non-negative including a genuine 0 — so that zero is exactly representable. That matters because padding and masks depend on zero staying zero. Second, the endpoints are pinned at exactly $-1$ and $+1$, so the block's absmax value survives quantization without error.

The payoff is measurable: at the same 4 bits per weight, NF4 gives lower perplexity than 4-bit integer or 4-bit float quantization of the same model. Same storage cost, better fidelity, from nothing but a better-chosen set of levels.

Double quantization: the constants are not free

Go back to the block-wise scheme and count honestly. Each block of 64 weights costs 64 four-bit indices plus one scaling constant $c$. If that constant is a 32-bit float, the true cost per weight is not 4 bits but

$$4 + \frac{32}{64} = 4.5\ \text{bits per parameter}$$

In words: one 32-bit constant is split 64 ways, so every weight quietly carries an extra half bit on top of its own four.

Half a bit of overhead, hiding in the bookkeeping. On a 65B model that is $0.5 \times 65 \times 10^9 / 8 = 4.06$ GB spent on constants — a real fraction of a 48 GB card, and the difference between fitting and not fitting.

Double quantization is the obvious move once you see the problem: the scaling constants are themselves just an array of numbers, so quantize them. QLoRA groups 256 of the FP32 constants at a time and stores them as 8-bit floats with their own second-level FP32 constant. The cost per weight becomes

$$4 + \frac{8}{64} + \frac{32}{64 \times 256} = 4 + 0.125 + 0.002 = 4.127\ \text{bits per parameter}$$

Term by term: 4 bits for the weight index, $8/64$ because one 8-bit first-level constant is amortized over its 64 weights, and $32/(64 \times 256)$ because one 32-bit second-level constant is amortized over $64 \times 256 = 16{,}384$ weights. The saving is $4.5 - 4.127 = 0.373$ bits per parameter, which on a 65B model is about 3 GB recovered — small as a fraction, decisive at the margin.

One detail explains why this is safe. Before quantizing the constants, their mean is subtracted so they are centred on zero; the constants are all positive absmax values, and 8-bit floats spend their precision symmetrically, so centring them first stops half the representable range from going to waste.

Paged optimizers: surviving the spikes

The third piece is not about compression at all. It is about the fact that GPU memory use during training is not flat — it spikes. Long sequences, an unlucky batch, or the recompute step of gradient checkpointing (trading compute for memory by recomputing activations in the backward pass instead of storing them) can briefly push demand over the card's capacity. When that happens the process dies with an out-of-memory error, and a day of training is gone.

QLoRA's paged optimizer uses NVIDIA's unified memory to give the optimizer states a fallback: when the GPU is under pressure, pages of optimizer state are evicted to CPU RAM automatically, and paged back in when the optimizer step needs them. It is the same idea as an operating system's swap file, applied to one specific allocation.

Note what it is and is not. It does not reduce steady-state memory, and it is not a way to train a model that does not fit. It converts an occasional fatal spike into an occasional slow step. That is a modest-sounding property and an essential one, because the alternative to "sometimes slower" is "sometimes crashed at hour 19."

Where the gradients actually go

This is the part that most often gets misread, so it is worth being precise. QLoRA has two different numeric formats doing two different jobs:

• a storage data type — NF4 — for the frozen base weights $W$, and
• a compute data type — BF16 — for everything the arithmetic touches.

Nothing is computed in 4 bits. For each matrix multiply, the 4-bit block needed is dequantized back to BF16 on the fly, used, and discarded. The forward pass through a QLoRA layer is

$$h = \mathrm{dequant}_{\text{NF4} \to \text{BF16}}(W_{\text{NF4}})\, x^{\text{BF16}} + B^{\text{BF16}} A^{\text{BF16}} x^{\text{BF16}}$$

which is LoRA's $h = Wx + BAx$ with one function wrapped around $W$.

Now the backward pass, in one sentence: gradients propagate through the dequantized 4-bit weights but are only accumulated into the adapters. Unpacking that, because the two halves are easy to conflate:

1. To compute the gradient with respect to a layer's input — which is what lets the error signal reach earlier layers at all — you need $W^\top$. So the chain rule genuinely runs through the quantized weights, dequantized as needed. Without this, nothing below the last layer could learn.
2. But $W$ is frozen. No gradient with respect to $W$ is ever stored, no optimizer state for $W$ is ever allocated, and $W_{\text{NF4}}$ is never written to. The only parameters with a $\partial L / \partial \theta$ that survives the step are $A$ and $B$, in BF16.

This is why the quantization does not have to be reversible in the way training weights must be. A 4-bit representation is far too coarse to accumulate a sequence of small gradient updates into — repeated round-tripping would quantize every update to zero. QLoRA never asks it to. The 4-bit tensor is read-only for the entire run, and all the delicate accumulation happens in the 16-bit adapter, which is small enough to afford full precision.

It also explains the headline empirical claim. QLoRA reports matching 16-bit full fine-tuning and 16-bit LoRA quality on instruction-tuning benchmarks, despite the base weights being 4-bit. The quantization error is a fixed, frozen perturbation of the pretrained model — and the adapter is trained on top of the perturbed model, so it is free to learn a correction for it. A fixed error you can train against is a much weaker constraint than a per-step rounding error.

The budget, recounted

Put the three pieces together and redo the arithmetic for 65B parameters on one card:

Two separate ideas did the work, and it is worth keeping them apart. LoRA killed the optimizer and gradient terms — the 650 GB. 4-bit quantization killed three quarters of the weight term — 130 GB down to 33.5 GB. Either alone leaves you off the card; together they land inside it.

The paper's demonstration of this was Guanaco, a family of models fine-tuned this way. The 65B member was trained on a single 48 GB GPU in about 24 hours, and the paper reports it reaching 99.3% of ChatGPT's score on the Vicuna benchmark as judged by GPT-4 — a figure worth treating as a directional result rather than a precise one, since LLM-as-judge benchmarks are noisy and favour verbose, well-formatted answers. The robust claim underneath it is the controlled comparison: at matched settings, QLoRA fine-tunes came out level with 16-bit fine-tunes, which is what makes the memory saving free rather than a trade.

Seeing it in code

In practice you never implement NF4 yourself. The bitsandbytes library provides the data type and the fused dequantize-and-multiply kernels; transformers and peft wire it up. The whole method is a config object and four lines:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

qcfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # the quantile table, not plain int4
    bnb_4bit_use_double_quant=True,        # quantize the scaling constants too
    bnb_4bit_compute_dtype=torch.bfloat16, # storage is 4-bit; math is 16-bit
)

model = AutoModelForCausalLM.from_pretrained("<base-model>", quantization_config=qcfg)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)

model = get_peft_model(model, LoraConfig(
    r=64, lora_alpha=16, lora_dropout=0.05, task_type="CAUSAL_LM",
    # QLoRA's finding: adapt *every* linear layer, not just attention q/v
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
))
model.print_trainable_parameters()   # typically well under 1% of parameters

Then use paged_adamw_32bit as the optimizer to get the paging behaviour, and train with an ordinary loop. Every line maps to a section above: bnb_4bit_quant_type is NF4, use_double_quant is the constants-of-constants trick, compute_dtype is the storage/compute split, target_modules is the ablation result, and the optimizer name is the spike insurance.

Pitfalls

It does not make inference faster. This is the single most common misreading, and it follows directly from the design. Every matrix multiply now pays a dequantization cost it did not pay before, and the arithmetic still happens in BF16 — so for compute-bound work like batched serving, 4-bit NF4 is typically slower per token than plain BF16, not faster. What 4 bits buys is that the model fits, and that memory-bandwidth-bound generation (small batch, one token at a time) has less to read. If throughput at scale is the goal, the right tools are inference-oriented quantization schemes with matching kernels, not the training-oriented one described here.

A QLoRA training step is slower than a 16-bit LoRA step. Same reason. You are trading time for memory; the trade is only worth making when 16-bit does not fit at all.

You cannot cleanly merge the adapter back into a 4-bit base. LoRA's nice property — fold $BA$ into $W$ after training and ship a normal model with zero inference overhead — needs $W$ in a format that can absorb a small additive change. Adding a BF16 $BA$ to an NF4 $W$ and re-quantizing loses much of what you trained. The standard route is to load the base in 16 bits, merge there, then quantize the merged result if you want it quantized — which means the machine that merges needs enough memory to hold the 16-bit model, even though the machine that trained it did not.

Adapt all the linear layers. The paper's ablations found that restricting LoRA to the attention projections left a measurable gap against full fine-tuning, and that the gap closed when every linear layer got an adapter. Rank mattered less than coverage. If a QLoRA run underperforms, the target_modules list is the first thing to check, not $r$.

Your baseline moved. The 4-bit model is not the model you started with — quantization perturbed it. If you want to know what the fine-tune contributed, evaluate the quantized, un-adapted model as the baseline. Comparing an adapted 4-bit model against an unquantized 16-bit original conflates two different effects.

The paging path is environment-specific. Paged optimizers depend on CUDA unified memory and enough host RAM to absorb the evicted pages. It is a Linux/NVIDIA feature, and on a machine with little spare system memory it cannot help.

How it's used in practice

The people who reach for QLoRA are mostly ML engineers adapting an openly available base model to their own data — internal documents, support transcripts, domain-specific instruction sets — on one or a handful of GPUs. The typical trigger is "16-bit LoRA ran out of memory," or "we want to know whether fine-tuning helps at all before renting a cluster."

The knobs you actually touch. The config in the code above is nearly all of it:

• BitsAndBytesConfig: load_in_4bit / bnb_4bit_quant_type="nf4" / bnb_4bit_use_double_quant / bnb_4bit_compute_dtype — the four settings that fix the storage and compute formats
• LoraConfig: r / lora_alpha / lora_dropout / target_modules — the shape of the adapter and where it goes
• On the training side, optim="paged_adamw_32bit" and gradient_checkpointing — the two things that get you over memory spikes
• After training, merge_and_unload() in peft — folds the adapter into the base, and as discussed above, should be run against a base loaded in 16 bits

The order of decisions. First estimate whether 16-bit LoRA fits; if it does, use it — each step is faster. Switch to QLoRA only when it does not. If results disappoint, confirm target_modules covers every linear layer before raising r. Don't guess peak memory: log torch.cuda.max_memory_allocated() for a few steps, then tune batch size and sequence length against the measured number.

Pitfalls that cause real incidents.

• Leaving out bnb_4bit_compute_dtype. If you don't set it, the library default is used, which may not be the BF16 you intended. Speed and numerical behaviour both change, so always set it explicitly.
• A GPU without BF16 support. BF16 assumes an Ampere-generation or newer NVIDIA GPU. On older cards you have to switch compute_dtype to something like FP16, and pay closer attention to numerical stability.
• Saving the adapter without recording how the base was loaded. What gets saved is just the small $A$ and $B$. To use them later you must load the same base model with the same quantization config; otherwise you are putting the adapter on top of a different model from the one it was trained against. Record the base revision and the contents of BitsAndBytesConfig alongside the adapter.
• Evaluating against the unquantized original. As in the pitfalls above, the right comparison is the quantized, un-adapted model.

Questions that come up in interviews and design reviews.

• "Why did training get slower with QLoRA?" → Every matmul dequantizes NF4 to BF16 first. You are trading time for memory.
• "If the weights are 4-bit, how do gradients flow?" → They pass through the dequantized weights to reach earlier layers, but accumulate only in the BF16 adapters. The 4-bit tensor is never written to.
• "What do you need to ship a trained adapter to production?" → Choose between merging on a machine that can hold the 16-bit base, or serving base plus adapter unmerged. The first costs memory at merge time; the second adds the adapter's compute at inference.

If you want to rebuild your intuition for low-rank approximation itself, SVD and Low-Rank Approximation is a good place to start.

Summary

• A 65B model is 130 GB of weights in BF16, and full fine-tuning needs roughly six times that once gradients and Adam states are counted
• LoRA removes the gradient and optimizer terms by freezing $W$ and training a low-rank update $\Delta W = BA$; it does not shrink $W$
• QLoRA shrinks $W$ to about 4.127 bits per parameter with NF4, a 16-level quantile table matched to the normal distribution that trained weights approximately follow, applied block-wise in blocks of 64
• Double quantization compresses the per-block scaling constants themselves, recovering 0.37 bits per parameter (3 GB at 65B)
• Paged optimizers evict optimizer state to CPU RAM during memory spikes, turning a crash into a slow step
• Storage is 4-bit, compute is 16-bit: weights are dequantized on the fly, gradients flow through them, and only the BF16 adapters are updated
• The quantization error is frozen, so the adapter can be trained to compensate for it — which is why 4-bit fine-tuning matches 16-bit quality
• It buys memory, not speed: inference and training steps are both somewhat slower than their 16-bit equivalents

If the low-rank half needs filling in, start with the LoRA paper explained; for the numeric formats underneath NF4, see Number Formats and Quantization Explained.

Check your understanding

1. QLoRA stores the frozen weights in 4 bits, yet claims to match 16-bit fine-tuning. Why doesn't the quantization error accumulate over thousands of training steps the way it would if you tried to train 4-bit weights directly?
2. With block size 64 and FP32 scaling constants, the real cost per parameter is 4.5 bits rather than 4. Where does the extra half bit come from, and what would happen to it if you raised the block size to 1024?
3. A colleague reports that after switching a fine-tune to QLoRA, their inference endpoint got slower. Is this a bug? Explain.
4. Why does NF4 use non-uniformly spaced levels, and what property of trained neural network weights makes that a win?
5. You trained a QLoRA adapter on a 70B base and now want a single merged 16-bit model to deploy. What do you need that the training machine did not need, and why?