pretraining

Definition

The first and largest training stage: run a self-supervised objective, usually next-token prediction, over a huge undifferentiated corpus to build general representations. It consumes most of the compute budget of a model programme, and the data mixture chosen here largely determines what the finished model knows. Everything after it — fine-tuning, preference optimisation — mostly shapes how that knowledge is expressed rather than adding to it.

Explainers using this term

• Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes