rectifiers

Definition

The family of activation functions that clip negative inputs, named after the electrical rectifier; ReLU is the canonical member. They displaced sigmoid-style activations largely because their gradient does not saturate for positive inputs, which is what made training genuinely deep networks practical. Variants differ in what happens below zero: Leaky ReLU keeps a small fixed slope, PReLU learns that slope, and GELU smooths the corner.

Explainers using this term

• Activation Functions from Scratch — Why Nonlinearity Is Non-NegotiableDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification