normalizer

Definition

The denominator that makes a set of scores sum to one — the partition function. In softmax it is the sum of exponentials over all entries, which is why attention naively needs the whole row before it can produce any output, and why streaming or tiled implementations carry a running maximum and running sum and rescale as they go. Distinct from normalization layers such as LayerNorm, which rescale activations rather than form a probability distribution.