normalization

Definition

Rescaling values so their distribution is well behaved. In deep nets it names a family of in-network layers — BatchNorm, LayerNorm, RMSNorm — that divide activations by statistics computed over some axis, stabilising gradients and permitting larger learning rates. Note the overload: the same word covers ordinary input preprocessing, but the interesting sense here is a trainable layer inside the model.

Explainers using this term

• A History of Normalization Layers — From BatchNorm to RMSNormBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift