gelus

Definition

Gaussian Error Linear Units: an activation that multiplies its input by the standard normal CDF evaluated at that input, giving a smooth, everywhere-differentiable curve instead of ReLU's hard kink at zero. Small negative values are damped rather than zeroed, which keeps gradients better behaved. GELU is the default nonlinearity in BERT- and GPT-style Transformers, and it is a different thing from GLU.