When the Transformer succeeded at language, an obvious question followed. Could it work on images too?

The answer given by the Vision Transformer (ViT) in 2020 was almost anticlimactically simple. Cut the image into square patches, say 16×16, treat each one as a single "word," and feed them straight into a Transformer.

No convolution at all. Flatten each patch into a vector, run it through a linear transformation to get an embedding, add positional information, and from there do exactly what a language model does. That alone, trained on enough data, beat CNNs on accuracy.

But there is a condition

This is where the "inductive bias" of Chapter 17 comes into play. A CNN has assumptions built into its very structure: that nearby pixels are related, and that shifting the position changes nothing. ViT has none of that. Every relationship has to be learned from the data.