residual

Definition

In a residual block the layer learns the difference from its input rather than the target directly, and a skip connection adds the untouched input back to the output. This is what made very deep networks trainable: gradients flow along the identity path instead of vanishing. The motif outlived the architecture that introduced it and now appears in essentially everything, Transformers included. Same root as the statistical residual, but here it names a structure, not an error term.