sparsity

Definition

The property that most entries are zero, or that only a small fraction of a model is active for any given input. It underlies weight pruning, sparse attention patterns, and mixture-of-experts routing alike — all ways to cut compute and memory. The standing caveat is that sparse does not imply fast: unstructured sparsity often gives no wall-clock gain, since GPUs only exploit patterns their hardware is built for.

Explainers using this term

• Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch TransformerSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity