laws

Definition

Empirical regularities, and in this field the word arrives almost exclusively as "scaling laws": the finding that loss falls as a smooth power law in model size, data, and compute. Their practical value is predictive — you can forecast what a training run will buy before committing the budget, and size the model and dataset accordingly. They are empirical fits over an observed range, not physical laws, and there is no guarantee they hold indefinitely.

Explainers using this term

• Paper Walkthrough — SMELT: Is Looping the Same Layers Twice Actually a Win When the Budget Is Matched?SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

In the textbook

• Scaling Laws and Their End