size

Definition

A word that means little without its qualifier: model size (parameter count), batch size, dataset size, context size. Scaling-law discussions turn on exactly which one is being varied, so conflating them is a common source of misreading. Mixture-of-experts models complicate matters further, since total and active parameter counts give two different, both legitimate, answers to how big the model is.