The GPT Lineage — Design Thinking from GPT-1 to Today

A generation-by-generation walk from GPT-1 to GPT-4o and the reasoning models, asking what changed and what deliberately did not. At the centre sits one machine that only ever predicts the next token; what moved was scale, the order of training, and how the output was disciplined.

The recipe stayed. Only the pot got bigger

When you want to improve a dish, you normally change the recipe — swap ingredients, add a step, adjust the heat. The history of GPT went the other way. The recipe barely changed; what changed, by orders of magnitude, was the size of the pot.

Put GPT-1 from 2018 next to a model running today and the skeleton is startlingly similar. Read text left to right, emit a probability for the next token. That single-purpose machine was made deeper, fed more text, and given more compute — year after year. The whole lineage is one long bet that scale buys capability, with the stake raised every round.

So generational change in GPT reads more accurately as a record of which axis was pushed than as a list of new features. This article walks each generation and asks what moved, and what deliberately stayed put.

The name is the design brief

GPT stands for Generative Pre-trained Transformer, and each word is a design commitment.

• Generative: emit the next word itself, not a classification label
• Pre-trained: before teaching the model any specific job, let it read enormous amounts of text and build a foundation
• Transformer: layers of attention stacked on each other — the mechanics are in Attention from Scratch

How unorthodox that combination was in 2018 shows up in the contrast. The mainstream approach was: for each task you care about, gather labelled data and train a dedicated model. One for translation, another for sentiment. GPT introduced a different order — read a mountain of text first with nothing being taught, then teach the job afterwards. Since nothing needs labelling, the study material is effectively unlimited.

One machine, common to every generation

Before walking the generations, let's pin down exactly what kind of machine this is. GPT treats the probability of a piece of text by factorising it like this.

Reading the symbols: $x_t$ is the $t$-th token (a word or fragment of one), $x_{<t}$ is everything before it, and $\prod$ means "multiply these together". Put in words, the formula says that the plausibility of a whole passage is the chance of each next token given everything read so far, multiplied all the way from start to finish.

What the model actually emits is a raw score (a logit) for every entry in the vocabulary. Softmax turns those into probabilities, and one token gets picked. Equation (1) has not changed since GPT-1. All that changed is how accurately those probabilities can be produced.

The picking step has a dial — temperature — and it does a lot to set the character of the output. Play with it below.

GPT-1 (2018): read first, install the job later

The first GPT was a 12-layer Transformer decoder with roughly 117 million parameters. It read BooksCorpus, a collection of unpublished books, and handled a context of 512 tokens. By current standards, a small model.

What the paper actually contributed was not scale but a procedure. Stage one: read books and learn nothing but next-token prediction. Stage two: fine-tune on labelled data for the task you want. That two-stage shape is what cemented "pre-train, then fine-tune" for language models.

The other move that paid off was input transformation. Tasks with different shapes — entailment, similarity, multiple choice — were rewritten as a single token sequence with delimiters, so the model body did not have to be rebuilt per task. The now-obvious idea that one architecture solves everything has its seed here.

Worth noting: GPT-1 still updated weights per task. Pre-training was only groundwork; wanting translation meant fine-tuning for translation. Removing that "install it later" step is the theme of the next two generations.