feedforward

Definition

Information flowing in one direction only, with no recurrence. Inside a Transformer the word usually names something specific — the position-wise feed-forward network in each block, which transforms every token independently while attention is what mixes information across tokens. A large share of a model's parameters sits in these layers, which is why mixture-of-experts and other efficiency techniques typically target the feed-forward block rather than attention.

Explainers using this term

• Weight Initialization and Regularization — What Lets Training Start, and What Keeps It GoingUnderstanding the Difficulty of Training Deep Feedforward Neural Networks