activation

Definition

The nonlinearity applied after a layer's linear transform — ReLU, GELU, SwiGLU and friends. Without one, stacking layers collapses into a single linear map, so it is the component that makes depth mean anything. The plural often refers instead to the values themselves, as in activation memory or activation checkpointing, where the concern is how many intermediate tensors must be kept alive during backpropagation.