Paper Deep Dive — ViT: Treating an Image Like a Sentence

A reading of the ViT paper (Dosovitskiy et al., 2020/2021) grounded strictly in its own text: the move of treating 16x16 patches as words, what the position-embedding ablation actually showed, the price of dropping the convolutional inductive bias, and how conditional the claim 'beats CNNs at scale' really is.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Laying out pixels one by one does not work

Self-attention compares every element of a sequence against every other. So why not turn an image into a sequence of pixels? The paper rules this out in the opening of its related work: applied naively, every pixel would attend to every other pixel, and the quadratic cost in the number of pixels does not scale to realistic input sizes. That is why the research of the day designed specialised attention patterns — local neighbourhoods, sparse approximations, one axis at a time — and those, the paper notes, "require complex engineering to be implemented efficiently on hardware accelerators" (§2). That is the starting point.

The move: cut the image into patches

What ViT does is simple. Reshape an image $\mathbf{x}\in\mathbb{R}^{H\times W\times C}$ into fixed-size $P\times P$ patches, flatten them, and map each to $D$ dimensions with a trainable linear projection. The patch count is $N=HW/P^{2}$, and that number is the effective sequence length the Transformer sees (§3.1).

Here $\mathbf{x}^{i}_{p}\mathbf{E}$ is the embedding of patch $i$, $\mathbf{x}_{\text{class}}$ is a learnable vector prepended to the sequence in the style of BERT's [class] token, and $\mathbf{E}_{pos}$ is the position embedding. From there a standard Transformer encoder runs unchanged, and the state of the leading token at the last layer becomes the image representation (§3.1).

Stated in words, the line says: lay the patches out in a row, put one extra slot at the front that belongs to no patch and exists only to accumulate a verdict about the whole image, and add to every slot a vector recording where in the picture that slot came from. $\mathbf{z}_{0}$ is that row, and it is what the encoder receives as its input.

One 16×16 block of pixels equals one word: that reading is what the title announces. The intent of staying as close as possible to the original Transformer is stated explicitly, and the payoff is that scalable NLP implementations transfer almost out of the box (§3). The notation ViT-L/16 means the Large variant with patch size 16, and since sequence length is inversely proportional to the square of the patch size, smaller patches cost more compute (§4.1).

The position embedding barely mattered

ViT uses standard learnable 1D position embeddings, on the grounds that more elaborate 2D-aware variants gave no significant gain (§3.1). The ablation in Appendix D.4 is blunt about it. On ImageNet 5-shot linear: no positional information 0.61382, 1-D 0.64206, 2-D 0.64001, relative 0.64032. The gap between having and not having them is large; the gap between encoding schemes is close to nothing (Table 8). The paper's guess is that because the encoder works on patches rather than pixels, the spatial grid is only about $14\times14$, and learning spatial relations at that resolution is equally easy for any of these schemes.

The visualisations in §4.5 close the loop: after training, nearby patches have similar position embeddings and a row-column structure appears. The model learned 2D topology on its own — which, as the paper puts it, explains why hand-crafted 2D-aware variants yield no improvement. For the general question of how position is injected, see positional encoding in Transformers.

What was given up: the inductive bias

In a CNN, locality, two-dimensional neighbourhood structure and translation equivariance are baked into every layer. In ViT only the MLP layers are local and translation-equivariant; the self-attention layers are global. Two-dimensional structure enters at exactly two points — cutting the image into patches at the start, and adjusting the position embeddings when the resolution changes — and everything else about spatial relations has to be learned from scratch (§3.1, "Inductive bias").

The consequence shows up in accuracy. Trained on a mid-sized dataset such as ImageNet without strong regularisation, these models land a few percentage points below ResNets of comparable size. The paper calls this outcome "expected" (§1).

The price is paid in data

Section 4.3 measures it two ways. First, by growing the pre-training set: ImageNet (1.3M images) → ImageNet-21k (14M) → JFT-300M (303M). Table 5 is unambiguous. Pre-trained on ImageNet alone, ViT-L/16 scores 76.53 and loses to ViT-B/16 at 77.91. On ImageNet-21k the order flips, 85.15 to 83.97; on JFT-300M the gap widens to 87.12 versus 84.15. A bigger model only pays for itself on bigger data.

Second, by training on random 9M/30M/90M/full subsets of JFT with no extra regularisation and identical hyperparameters. The finding: ViT overfits more than ResNets of comparable computational cost on smaller datasets. The convolutional inductive bias is useful when data is scarce, but with enough data, learning the relevant patterns directly is sufficient — even beneficial. That is the paper's conclusion.

"Beats CNNs" comes with conditions

Table 2 holds the headline results. Pre-trained on JFT-300M, ViT-H/14 reaches 88.55 on ImageNet, 90.72 on ImageNet-ReaL, 94.55 on CIFAR-100 and 77.63 on the 19-task VTAB suite. BiT-L (ResNet152x4) gets 87.54 / 90.54 / 93.51 / 76.29, and Noisy Student reaches 88.4 on ImageNet (§4.2). What matters more than the margins is the pre-training cost: 2.5k TPUv3-core-days for ViT-H/14 and 0.68k for ViT-L/16, against 9.9k for BiT-L and 12.3k for Noisy Student.

The paper immediately qualifies this. Pre-training efficiency depends not only on the architecture but on the training schedule, the optimiser and weight decay (§4.2). The controlled comparison is in §4.4: transferring from JFT-300M, ViT reaches the same performance with roughly 2–4× less compute. Hybrids, which feed ResNet feature maps into ViT, edge out pure ViT at small compute budgets, but the difference vanishes as models grow.

What happens inside

Section 4.5 measures "attention distance" — the mean distance in image space between pixels, weighted by attention, analogous to receptive field size in a CNN. Some heads already attend to most of the image in the lowest layers. Other heads in those low layers keep consistently small attention distances, and this localised attention is far less pronounced in hybrid models; the paper suggests it may be serving the same purpose as early convolutional layers (Appendix D.7). For the mechanism itself, see self-attention from scratch.

Limits and open questions

Stated by the paper itself:

• Detection and segmentation were not attempted. The conclusion lists them as remaining challenges (§5).
• Self-supervision is preliminary. Masked patch prediction gets ViT-B/16 to 79.9% — a 2% gain over training from scratch, but still 4% behind large-scale supervised pre-training (§4.6).
• The head choice is entangled with the learning rate. An early attempt to replace the [class] token with global average pooling plus a linear classifier "performed very poorly", yet the difference was fully explained by the need for a different learning rate (Appendix D.3).
• Shape scaling is heuristic. Depth helps most, with diminishing returns after 16 layers, and width helps least (Appendix D.2). Beyond "scaling all dimensions proportionally is robust", there is no principle here.

Note as of writing (2026) — an author's remark, not a claim of the paper: the comparison points are BiT and Noisy Student as of 2020–2021, and every number above is this paper's own. Results from later training recipes or distilled variants are not included.

How this shows up on the job

An ML engineer meets ViT mostly when fine-tuning published pre-trained weights on their own data — visual inspection, medical imaging, satellite imagery. Given that the paper's claim only holds with large-scale pre-training behind it, that is also the correct way to use it.

The knobs are largely fixed. Patch size (/16 or /32), input resolution, and interpolation of the position embeddings. The paper states that fine-tuning at a higher resolution than pre-training is often beneficial: keep the patch size, let the sequence get longer, and interpolate the pre-trained position embeddings in 2D according to their location in the original image (§3.2). Fine-tuning runs with SGD, momentum 0.9, batch 512, no weight decay, gradient clipping at global norm 1, resolution 384, and a small learning-rate grid search (Table 4). Pre-training uses Adam, batch 4096, weight decay 0.1 (Appendix B.1).

Two ways to get hurt. First, that position-embedding interpolation. Change the resolution and the patch count changes with it; skip the interpolation and accuracy quietly drops with no error anywhere. Second, misjudging data volume. Pick Large over Base with a few tens of thousands of images and you reproduce Table 5 exactly. "Bigger model, better results" is a statement about the pre-training data, not about your downstream set.

Questions that come up in interviews and design reviews. "Why does ViT need so much data?" — it discards locality and translation equivariance and learns spatial relations from scratch (§3.1). "What happens to compute if you go from patch 16 to patch 8?" — sequence length is inversely proportional to the square of the patch size, so 4× the tokens (§4.1), and all-pairs attention scales with the square of that, so roughly 16× for the attention part (the second half is my own arithmetic from the statement in §2). For the CNN side of the comparison, image classification from the ground up has the background.

Summary

• ViT cuts an image into patches, projects them linearly and feeds a standard Transformer encoder. The only image-specific design is the initial patch extraction and the position-embedding adjustment on resolution change (§3.1, §3.2)
• Having position embeddings matters; how you encode them barely does (Appendix D.4)
• The price of dropping the inductive bias is data. At ImageNet scale, Large loses to Base (§4.3)
• "Beats CNNs" is conditional on large-scale pre-training. The compute needed for equal performance is roughly 2–4× lower (§4.2, §4.4)

Original paper: Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" (arXiv:2010.11929, 2020 / v2 2021)