A History of Normalization Layers — From BatchNorm to RMSNorm

The layer that made deep learning actually deep, explained from zero. The internal-covariate-shift controversy behind BatchNorm, why LayerNorm threw away the batch axis, and why every modern LLM converged on RMSNorm.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

The wall you hit when you stack layers

Until around 2015, neural networks had a strange ceiling. In theory, more layers meant more expressive power — but stack ten or twenty and training simply stopped working. Turn the learning rate up and the loss diverged; turn it down and the loss barely moved.

One culprit was the scale of the numbers flowing between layers. Picture an assembly line where the parts arriving from upstream are twice as large one day and a tenth the size the next: the downstream station has to recalibrate its tools every morning. The same thing happens during training. Every time an earlier layer's weights update, the layers above it have to relearn starting from "how big are the incoming numbers, roughly?" A normalization layer is the jig you bolt onto a layer's output to cut that loop. This article traces the line from BatchNorm in 2015 to RMSNorm, which nearly every LLM now uses — with the reasons each replacement happened.

When values overflow, gradients vanish

Why does scale matter so much? The fastest way to see it is to move an activation function around yourself.

On those flat stretches the slope is essentially zero. Learning only happens by nudging weights along the gradient, so a layer whose gradient hits zero stops learning. Worse, backpropagation carries gradients backward by multiplication, so a single near-zero layer drags every layer beneath it down with it (see Backpropagation from Scratch). Normalization exists to push a layer's inputs back away from that cliff edge.

Standardize to mean 0, variance 1 — then undo it

Every normalization layer, without exception, is these two expressions.

Here $x$ is the value being normalized, $\mu$ is the mean of the set it belongs to, $\sigma^2$ is the variance, and $\varepsilon$ is a small constant that prevents division by zero. $\gamma$ (gamma) and $\beta$ (beta) are learned parameters: how far to stretch the result back out, and how far to shift it. Equation (1) says, in plain terms: flatten everything to mean 0 and spread 1, then let training stretch it back out by exactly as much as it needs.

Why undo what you just did? To protect expressiveness. If every layer's output is forced to mean 0 and variance 1, the model can no longer carry information in the magnitude of its values. With $\gamma$ and $\beta$ available, the model can learn $\gamma=\sigma$ and $\beta=\mu$ and cancel the normalization entirely when that is what it wants.

Everything above is shared by all three methods. What separates BatchNorm, LayerNorm and RMSNorm is exactly one thing: which set of values you compute $\mu$ and $\sigma$ over.

BatchNorm: average across the whole batch

BatchNorm (Ioffe & Szegedy, 2015) takes its statistics along the sample axis of a mini-batch. In a convolutional layer, that means collapsing every image and every pixel in the batch into a single mean and a single variance per channel. Which implies something odd: how one particular image gets transformed depends on the other images that happened to land in the same batch. That coupling is where BatchNorm's power comes from — and it is the root of every headache that follows.

At inference time the batch might contain a single item, so batch statistics are unavailable. BatchNorm therefore accumulates a running average of the mean and variance during training and uses those at inference. The formula itself differs between training and inference — which is precisely why you have to flip between model.train() and model.eval() in PyTorch.

The "internal covariate shift" controversy

The original paper explained BatchNorm's effectiveness as a reduction in internal covariate shift (ICS) — the phenomenon where the distribution of a layer's inputs keeps moving as the layers below it update. The claim is right there in the title: "Reducing Internal Covariate Shift." It was intuitive, memorable, and became the textbook account for years.

Then in 2018, Santurkar et al.'s "How Does Batch Normalization Help Optimization?" attacked it head-on. They deliberately injected noise immediately after the BatchNorm layers — noise with non-zero mean, non-unit variance, and a distribution that changed at every step. By the definition of ICS this should have been the worst possible condition, yet the network trained about as fast as a standard BatchNorm network. If you can crank ICS up on purpose without hurting performance, reducing ICS cannot be the explanation.