scale

Definition

In this field, the size of the system — parameters, training tokens, and compute — as in "at scale" or "scaling up". Its significance is empirical: performance improves predictably as these quantities grow, and that observation has driven the strategy of the last several years of model development. The everyday senses of scale, meaning a measuring scale or a ratio, are rarely what is meant here.

Explainers using this term

• Paper Walkthrough: X-AuT — Pull Audio-Encoder Layers Out Two at a Time, Then Backfill from a Bigger TeacherX-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
• Paper Walkthrough: Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training — training the draft alongside a 122B policy at 256K tokensOnline Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
• Paper Walkthrough — RoboTok: Mining the Web for Demonstrations That Move Like YoursRoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning