Encoder or Decoder — The Fork in the Road Between BERT and GPT

One masked triangle in the attention table is what separated BERT from GPT. This piece works through bidirectional versus autoregressive with the equations and an interactive figure, then asks why generation won and where encoders are still the first choice.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Twins from the same blueprint

Within about a year of the Transformer blueprint appearing in 2017, two models came out that used it almost verbatim: BERT and GPT. Same parts, same way of stacking them — and yet completely different uses, and completely different fates.

What separated them was a single design decision: how much of the sentence is each word allowed to see? That one choice, and almost nothing else, turned one into a reading machine and the other into a writing machine.

Think of BERT as a proofreader. The whole manuscript is spread out on the desk, and a smudged character gets reconstructed from what sits on either side of it. What a proofreader does not do is write a page from scratch. GPT is a storyteller, choosing the next word as they speak, so the visible range is always and only "everything before me."

The intuition: what you can see decides what you can be trained on

This is not a matter of taste. Once you fix the visible range, the shape of the training task is fixed for you.

The moment you decide, as GPT does, that a position sees only what precedes it, the training task becomes free. At every position, the next word is the label. Nobody has to annotate anything; text alone generates an unlimited supply of problems. That is autoregressive modelling.

Decide instead, as BERT does, that everything is visible, and you have a problem. Ask the model to predict the next word and it can already see it. There is nothing to learn from an open-book exam. So BERT introduced a deliberately artificial step: hide some of the words, then ask for them back. That is masked language modelling (MLM), and the original paper hides 15% of the input tokens.

Going bidirectional costs you an extra masking step; going autoregressive costs you the right to look backwards from the future. Both sit on top of a trade-off.

The mechanism: one line drawn across the attention table

As covered in Attention from scratch, self-attention builds an $N \times N$ table of scores for "how much does word $i$ care about word $j$", where $N$ is the number of tokens.

Here $q_i$ is word $i$'s "question" vector, $k_j$ is word $j$'s "name tag" vector, and $d_k$ is their dimension. Put in words, it says: measure how similar $i$'s question is to $j$'s name tag, then flatten the scale by the square root of the dimension.

The fork is entirely in how that table is treated.

GPT adds this one line, which says: replace every score that points into the future ($j > i$) with $-\infty$. Because $e^{-\infty}=0$, softmax turns those weights into exactly zero and no future word gets mixed in. Blacking out the upper-right triangle this way is the causal mask. BERT never adds it and uses the full table. The layer structure, the residual connections, the feed-forward blocks are shared, so structurally this is nearly the whole difference.

Where it really bites is the training objective. BERT takes its loss only at the hidden positions.

$M$ is the set of hidden positions and $x_{\setminus M}$ is everything left visible. Put in words: recover each hidden word using all of the remaining context, and score yourself only on the hidden spots.

GPT takes its loss at every position.

$x_{<i}$ is every word before $i$ — which says: predict each word from its left context only, and every single position counts as a question.

Try flipping the "causal mask" switch in the figure below. Switched on is the world GPT sees; switched off is the world BERT sees.