continued

Definition

Resuming training on an already-pretrained model rather than starting over. Continued pretraining keeps the original objective but feeds a new corpus, typically to adapt a general model to a domain or a language, at a data scale closer to pretraining than to fine-tuning. The recurring problem is catastrophic forgetting — gains on the new distribution bought with losses on the old — which is why data mixing ratios and replay are usually the substance.

Explainers using this term

• Paper Explained: Beyond Data Scaling — Why the Backbone, Not the Trajectory Count, Decides Your VLA (VLAct)Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models