scaling

Definition

How performance changes as you increase parameters, data, or compute. Empirical scaling laws — loss falling as a smooth power law in each resource — turned model building into a budgeting problem: given fixed compute, how much should go to model size versus training tokens. Distinct from the everyday sense of resizing, and from scalability as an engineering property.

Explainers using this term

• GE-Act 2.0 Paper Explained: Pretraining and Scaling a World-Action Model for Robotic ManipulationGE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
• Paper Walkthrough: Puffin-World — A World Model That Remembers Which Way Is UpPuffin-World: Scaling a Unified Multimodal Model with Native 3D World States
• SMELT explained (arXiv:2609.01343) — compute-matched loopingSMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

In the textbook

• Scaling Laws and Their End
• What Happens When You Make It Bigger — Scaling and Pre-training