pre-training

Definition

The stage where a model is trained on large amounts of general data before it is pointed at any specific task, usually with a self-supervised objective such as next-token or masked-token prediction. This is where most of the compute budget goes in modern large models; later adaptation is comparatively cheap. Contrast fine-tuning, which is the task-specific step that comes after.

Explainers using this term

• Paper Walkthrough: ZimaBlue — Turning 120,000 Hours of Egocentric Video into Robot SkillZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training
• Paper Explained: Beyond Data Scaling — Why the Backbone, Not the Trajectory Count, Decides Your VLA (VLAct)Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models
• Encoder or Decoder — The Fork in the Road Between BERT and GPTBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding