bert

Definition

A pre-trained language model from Google built on a Transformer encoder, trained by masking out tokens and predicting them from context on both sides at once. Its historical importance is that it made the pre-train-then-fine-tune recipe the default across NLP, with one backbone adapted to many downstream tasks. Unlike decoder-only GPT-style models it is not built for open-ended generation; it is an encoder aimed at understanding, classification, and retrieval.

Explainers using this term

• Encoder or Decoder — The Fork in the Road Between BERT and GPTBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding