pre-train

Definition

To pre-train is to run a first, general-purpose training phase on large data so that later fine-tuning starts from useful representations instead of random weights. Pre-training objectives are typically self-supervised — predict the next token, reconstruct a masked patch — precisely because they need no labels and can therefore consume web-scale corpora. The cost profile is inverted relative to fine-tuning: enormously expensive once, then cheap to reuse indefinitely.