Paper Walkthrough: Normalized Low-Rank Adaptation — Why Normalizing LoRA's Entry Matrix Works

Rescaling LoRA's down-projection so every column has unit length improves convergence, stability and forgetting resistance at zero extra cost. A ground-up reading of Normalized Low-Rank Adaptation (NoRA) through the lens of a hidden preconditioner.

Normalized Low-Rank Adaptation

The entry matrix decides LoRA's first step

The paper walked through here is titled "Normalized Low-Rank Adaptation" (Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu / arXiv:2608.31036, posted 31 August 2026). The method it proposes is called NoRA.

The claim, in the paper's own framing: LoRA is everywhere, yet how to regularize its training dynamics for stable and effective optimization stays underexplored. Because LoRA initializes the up-projection $\bm{B}$ to zero, early optimization is governed almost entirely by the down-projection $\bm{A}$. NoRA normalizes $\bm{A}$ during training. Applying the same normalization only at initialization also improves plain LoRA. Across pretraining, supervised finetuning (SFT) and reinforcement learning this accelerates convergence, improves performance and stability, and mitigates catastrophic forgetting — with no extra trainable parameters and no inference-time computation (Abstract).

A metaphor: a mixing desk with faders set at random

$\bm{A}$ is the entry point that pushes each input dimension into a narrow $r$-dimensional latent space — one fader per input channel on a mixing console. Standard LoRA sets those faders from random numbers, and because the convention is to sample with small variance, they tend to sit uniformly low as well. Start recording in that state and the channels that come through loudly get tuned quickly, while the muted ones barely register. In other words, how fast each input dimension learns is decided by the initialization seed, not the data or the task. NoRA levels every fader before recording starts.

Background: what $\bm{B}=\bm{0}$ costs

LoRA writes the update to a pretrained weight $\bm{W}_0$ as a product of two thin matrices (§2).

In plain words: express the whole change in the weight as one skinny matrix times another. $\bm{A}\in\mathbb{R}^{r\times k}$ is the down-projection squeezing $k$ input dimensions into $r$; $\bm{B}\in\mathbb{R}^{d\times r}$ is the up-projection back to $d$ output dimensions; $r$ is a rank far below $d$ or $k$; $\alpha$ is a scaling factor.

What matters is the initialization $\bm{B}^{(0)}=\bm{0}$. Since $\Delta\bm{W}$ starts at zero, the model at step 0 behaves exactly like the pretrained one — safe, but with a side effect. Writing $\bm{G}=\partial\mathcal{L}/\partial\bm{W}$ for the gradient full finetuning would follow (§2):

Read simply: at the first moment of training, $\bm{A}$ receives no gradient. With $\bm{B}$ at zero, moving $\bm{A}$ changes no output and therefore no loss. $\bm{A}$ is left at its initial value, and LoRA's early optimization depends entirely on the random features induced by $\bm{A}^{(0)}$ (§2). That is the paper's starting point.

The mechanism: unit-length columns along the rank axis

The idea comes from Multi-head Latent Attention (MLA) (§3.1). MLA normalizes the latent representation in its bottleneck, and that was observed to stabilize training and speed up convergence: $\bm{\phi}(\bm{x})=\text{Norm}(\bm{A}\bm{x})$ normalizes the feature after projection.

Dropping that straight into LoRA creates a problem. The normalization depends on the input, so $\bm{B}\cdot\text{Norm}(\bm{A}\bm{x})$ is nonlinear in $\bm{x}$ and can no longer be folded into $\bm{W}_0$ after training (§3.1). Since most of LoRA's practical value is "merge it and pay nothing at inference," that is a bad trade.

So the paper moves the normalization from the feature to the projection matrix itself. Column $j$ of $\bm{A}$, written $\bm{a}_j\in\mathbb{R}^{r}$, describes how input coordinate $j$ is sent into the latent space; normalize it along the rank axis (§3.1).

All it does is divide each column of $\bm{A}$ by its own length ($\epsilon$ guards against dividing by zero).