Weight Initialization and Regularization — What Lets Training Start, and What Keeps It Going

The same architecture will train or refuse to train depending on nothing more than how large the random numbers in its weights were. Starting from how variance propagates, this piece derives Xavier and He initialization, then turns to weight decay and dropout as the conditions that keep training going — with interactive figures, PyTorch code and the mistakes that actually cost people days.

Understanding the Difficulty of Training Deep Feedforward Neural Networks

Same blueprint, and only one of them moves

Same depth, same activation function, same learning rate, same data. The only difference is the size of the random numbers put into the weights before training started. And with that alone, one run's loss falls steadily while the other's does not budge over the first thousand steps. Anyone who has trained a network has seen this at least once.

The closest analogy is audio. Picture a stack of amplifiers wired in series. If each stage multiplies the signal by 1.1, ten stages give about 2.6×, fifty give about 117×, and a hundred push past 10,000×. If each stage instead multiplies by 0.9, the signal after a hundred stages is less than one thirty-thousandth of what went in. A small change in the per-stage gain becomes an exponent in the number of stages. Either it screams with feedback or you hear nothing at all — only a narrow band around 1.0 is "music".

The layers of a neural network are exactly these amplifiers, and the initial weights set each stage's gain. The first half of this article is about initialization, the condition that lets training start. The second half is about regularization, the condition that keeps it going.

Magnitude gets multiplied at every layer

What a single neuron does is multiply its inputs by weights and add them up.

$x_i$ is the $i$-th input arriving from the previous layer, $w_i$ is the weight it gets multiplied by, and $n$ is the number of lines coming into this layer. What equation (1) says in words is simply: take every incoming number, scale it by a coefficient called a weight, and add all of them together. That is the whole operation.

The interesting part is the spread of that sum. Initialize the weights $w$ with mean-zero random numbers, assume they are independent of the input $x$, and elementary statistics gives the spread of the output.

Variance is a measure of "how spread out the values are" — think of it as the square of the width of the spread. Equation (2) states that the spread of the output is the spread of the input multiplied by (number of inputs × spread of the weights).

The same thing in words: how big you make a single weight, and how many inputs there are — those two numbers alone decide what factor this layer multiplies the signal by. With 100 inputs and a weight spread of 0.01, the factor is exactly 1.0. Keep the weights the same size and grow the layer to 10,000 inputs, and the factor becomes 100.

That $n \cdot \mathrm{Var}(w)$ is the amplifier gain from earlier. Above 1 and values swell with every layer; below 1 and they shrink. The deeper the network, the more that difference opens up exponentially.

This is precisely what made deep learning hard for so long. Swell, and you get exploding gradients and a loss of NaN; shrink, and you get vanishing gradients with updates rounded to zero. How backpropagation carries gradients as a per-layer product is covered in backpropagation from scratch, but the premise needed here is one line: anything carried by multiplication either dies exponentially or explodes exponentially.

On the vanishing side, the activation function makes it worse

What happens when values grow too large is obvious the moment you move an activation function yourself.

On the flat parts the slope is essentially zero. Learning only happens by moving weights along the gradient, so a layer that lands there stops moving. Worse, once one layer returns a near-zero gradient, every layer beneath it is dragged down with it. The intuition that "larger initial values mean a stronger signal and therefore faster learning" is betrayed by activation saturation.

The design condition: keep the variance unchanged

The condition is then obvious. Make the spread of the values survive a pass through the layer unchanged. Setting $\mathrm{Var}(y) = \mathrm{Var}(x)$ in equation (2) gives

$n_{\text{in}}$ is the number of lines entering the layer (fan-in). Equation (3) says: the more inputs a layer has, the smaller each individual weight should be. Use random numbers of the same size for a layer fed by 100 lines and one fed by 10,000, and the second one's output is ten times as spread out.

Put in words, it says to make the typical size of one weight — its standard deviation — around $1/\sqrt{n_{\text{in}}}$. For a layer with 512 inputs that is about 0.044; with 2048 inputs, about 0.022. Which carries an unremarkable but important consequence: if layers differ in fan-in, the random numbers you put into them should differ too.

There is a second condition, though. Preserving the spread of the values in the forward pass does not guarantee preserving the spread of the gradients in the backward pass. Gradients flow the other way, so that condition uses the number of outgoing lines $n_{\text{out}}$ (fan-out) and reads $\mathrm{Var}(w) = 1/n_{\text{out}}$. Both can hold at once only when $n_{\text{in}} = n_{\text{out}}$.

Xavier initialization: a compromise between two conditions

The answer Glorot and Bengio published in 2010 was a compromise — split the difference between the two conditions.

Equation (4) is Xavier initialization (named after the first author; also called Glorot initialization). Stated in words: average the number of ways in and the number of ways out, and use the reciprocal of that average as the variance of the weights — $2/(n_{\text{in}}+n_{\text{out}})$ is exactly the reciprocal of $(n_{\text{in}}+n_{\text{out}})/2$. Sitting halfway between the forward condition and the backward condition breaks neither badly. Implementations often use the uniform variant instead, drawing uniformly from $\pm\sqrt{6/(n_{\text{in}}+n_{\text{out}})}$ — the 6 falls out of the fact that a uniform distribution's variance is one twelfth of the square of its width.

What that paper contributed at the time was less the formula than the viewpoint: initial values are not something you guess at, they are something you derive from how variance propagates. Numbers that had been set by convention — "normal random numbers times 0.01" — finally had a reason behind them.

He initialization: ReLU discards half

Xavier initialization assumes sigmoid or tanh. Those are nearly linear around the origin, which licenses the implicit assumption that passing through the activation does not change the spread much.

With ReLU that assumption collapses. ReLU zeroes every negative input, so feeding it a mean-zero distribution kills exactly half of it, and roughly halves the variance too (see activation functions from scratch).

The correction He et al. published in 2015 was to cancel that "half" in advance.