Paper Deep Dive — ViT: Treating an Image Like a Sentence

A reading of the ViT paper (Dosovitskiy et al., 2020/2021) grounded strictly in its own text: the move of treating 16x16 patches as words, what the position-embedding ablation actually showed, the price of dropping the convolutional inductive bias, and how conditional the claim 'beats CNNs at scale' really is.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Laying out pixels one by one does not work

Self-attention compares every element of a sequence against every other. So why not turn an image into a sequence of pixels? The paper rules this out in the opening of its related work: applied naively, every pixel would attend to every other pixel, and the quadratic cost in the number of pixels does not scale to realistic input sizes. That is why the research of the day designed specialised attention patterns — local neighbourhoods, sparse approximations, one axis at a time — and those, the paper notes, "require complex engineering to be implemented efficiently on hardware accelerators" (§2). That is the starting point.

The move: cut the image into patches

What ViT does is simple. Reshape an image $\mathbf{x}\in\mathbb{R}^{H\times W\times C}$ into fixed-size $P\times P$ patches, flatten them, and map each to $D$ dimensions with a trainable linear projection. The patch count is $N=HW/P^{2}$, and that number is the effective sequence length the Transformer sees (§3.1).

Here $\mathbf{x}^{i}_{p}\mathbf{E}$ is the embedding of patch $i$, $\mathbf{x}_{\text{class}}$ is a learnable vector prepended to the sequence in the style of BERT's [class] token, and $\mathbf{E}_{pos}$ is the position embedding. From there a standard Transformer encoder runs unchanged, and the state of the leading token at the last layer becomes the image representation (§3.1).

Stated in words, the line says: lay the patches out in a row, put one extra slot at the front that belongs to no patch and exists only to accumulate a verdict about the whole image, and add to every slot a vector recording where in the picture that slot came from. $\mathbf{z}_{0}$ is that row, and it is what the encoder receives as its input.

One 16×16 block of pixels equals one word: that reading is what the title announces. The intent of staying as close as possible to the original Transformer is stated explicitly, and the payoff is that scalable NLP implementations transfer almost out of the box (§3). The notation ViT-L/16 means the Large variant with patch size 16, and since sequence length is inversely proportional to the square of the patch size, smaller patches cost more compute (§4.1).

The position embedding barely mattered

ViT uses standard learnable 1D position embeddings, on the grounds that more elaborate 2D-aware variants gave no significant gain (§3.1). The ablation in Appendix D.4 is blunt about it. On ImageNet 5-shot linear: no positional information 0.61382, 1-D 0.64206, 2-D 0.64001, relative 0.64032. The gap between having and not having them is large; the gap between encoding schemes is close to nothing (Table 8). The paper's guess is that because the encoder works on patches rather than pixels, the spatial grid is only about $14\times14$, and learning spatial relations at that resolution is equally easy for any of these schemes.

The visualisations in §4.5 close the loop: after training, nearby patches have similar position embeddings and a row-column structure appears. The model learned 2D topology on its own — which, as the paper puts it, explains why hand-crafted 2D-aware variants yield no improvement. For the general question of how position is injected, see positional encoding in Transformers.