autoencoding

Definition

Training a model to reconstruct its own input, usually through a bottleneck. In language modelling it names the BERT-style masked objective — corrupt part of the input, predict what was removed — which gives the model bidirectional context and representations that transfer well to classification and retrieval. The contrast is with autoregressive training, which predicts the next token left to right; autoencoding models see more context per token but cannot generate fluently by construction.

Explainers using this term

• Paper Walkthrough: AURORA-LM — A Diffusion Language Model That Keeps Its Text Latent Full-Width Instead of Compressing ItAURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling