trillion

Definition

A scale marker for parameter counts and training-token counts at the frontier. Read it carefully in Mixture-of-Experts papers: total parameters can reach the trillions while only a fraction is activated per token, so a trillion-parameter MoE and a trillion-parameter dense model have nothing like the same serving cost. Token counts are the more honest comparison, since every training token is genuinely processed.

Explainers using this term

• Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch TransformerSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity