electra

Definition

A pretraining method in which a small generator replaces some input tokens and a discriminator predicts, for every position, whether that token is original or substituted. Because the loss applies at all positions rather than the roughly 15% that masked language modelling supervises, it reaches a given quality with substantially less compute than BERT. The generator–discriminator setup resembles a GAN but is not adversarial: both are trained by maximum likelihood, and the generator is discarded afterwards.