switch

Definition

Selecting which path or component handles the input. In mixture-of-experts work it names the routing style popularised by Switch Transformer, where each token is sent to exactly one expert rather than being blended across several — simpler to implement and cheaper per token, at the cost of a harder load-balancing problem. The name borrows from network switches: pick one destination and forward.

Explainers using this term

• Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch TransformerSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity