scaling

Definition

How performance changes as you increase parameters, data, or compute. Empirical scaling laws — loss falling as a smooth power law in each resource — turned model building into a budgeting problem: given fixed compute, how much should go to model size versus training tokens. Distinct from the everyday sense of resizing, and from scalability as an engineering property.

Explainers using this term

• Paper Walkthrough: Puffin-World — A World Model That Remembers Which Way Is UpPuffin-World: Scaling a Unified Multimodal Model with Native 3D World States
• Paper Walkthrough — SMELT: Is Looping the Same Layers Twice Actually a Win When the Budget Is Matched?SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
• Paper walkthrough: CyberFactory — turning wild CVEs into runnable training problemsCyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild

In the textbook

• Scaling Laws and Their End