glu

Definition

Gated Linear Unit: project the input twice, pass one branch through a nonlinearity, and multiply it elementwise into the other, so the input itself decides which components get through. Variants that swap the gating nonlinearity — SwiGLU, GEGLU — are now standard in the feed-forward block of large Transformers, having displaced plain ReLU or GELU. Note it costs a third projection matrix, so hidden width is usually reduced to compensate.