self-supervised

Definition

Building the training signal out of the data itself — masking part of an input and predicting it from the rest, or predicting the next token. The loss looks exactly like supervised learning; only the source of the target differs. Because no annotation is required, it is what made pre-training on web-scale corpora feasible, and it underlies essentially every foundation model.

Explainers using this term

• Paper Review: S²VOPD — Don't Strengthen the Teacher, Blur the Student's ViewSelf-Supervised Visual On-Policy Distillation