From here the story moves over to language processing. And in the end it comes back to images.

For a long time the dominant models for text were RNNs (recurrent neural networks). They read the words one at a time, in order, updating an internal state as they go. But they had two weaknesses. Because the processing is sequential it cannot be parallelized, so training is slow. And relationships between words far apart from each other are hard to hold onto.

The idea behind attention

What the 2017 paper "Attention Is All You Need" put forward was a structure that stops processing in order and instead has every word refer directly to every other word.