bidirectional

Definition

Conditioning on context from both sides of a position rather than only what precedes it — the B in BERT. It is available to encoders trained with a masked objective, since nothing is being produced left to right, and it generally yields stronger representations for understanding tasks. Autoregressive decoders cannot use it: seeing later tokens while predicting the next one would leak the answer.

Explainers using this term

• Encoder or Decoder — The Fork in the Road Between BERT and GPTBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding