masked

Definition

Hiding part of the input and training the model to reconstruct it from what remains — masked language modeling for text, masked autoencoding for images. It is a self-supervised objective: the label comes from the data itself, so unlabeled corpora suffice. Unlike autoregressive prediction, which sees only the left context, a masked objective lets the model condition on both sides of the gap, which suits representation learning more than generation.