Paper Deep Dive — CLIP: Putting Words and Images on One Map

A reading of the CLIP paper (Radford et al., 2021) grounded strictly in its own text: how the in-batch positives and negatives work, why zero-shot classification is really a hypernetwork trick, the five points that prompt wording is worth, and the long list of limits the authors wrote themselves.

Learning Transferable Visual Models From Natural Language Supervision

If it does not fit in one of a thousand boxes, it is invisible

State-of-the-art vision systems are trained to predict a fixed set of predetermined categories. That form of supervision limits their generality, because specifying any other visual concept means collecting more labelled data. That is where the paper opens (Abstract).

The weakly supervised work that came before sat in the same cage. The paper points out that those approaches carefully design — and in the process limit — their supervision to 1000 classes, or 18291 classes. They also predict through a static softmax classifier with no mechanism for dynamic outputs, which severely curtails flexibility and caps zero-shot ability (§1). Meanwhile the attempts to learn directly from natural language performed poorly: Visual N-Grams reached 11.5% zero-shot on ImageNet (§1).

So the authors collect 400 million (image, text) pairs from publicly available sources on the internet, searching for text containing one of 500,000 queries (words appearing at least 100 times in English Wikipedia, WordNet synsets and so on) and class-balancing with up to 20,000 pairs per query. They call it WIT, for WebImageText (§2.2).

Predicting the caption was the long way round

The first approach jointly trained an image CNN and a text transformer to predict the caption. Efficiency never came. Figure 2 shows the measurement: a 63-million-parameter Transformer language model — already using twice the compute of its ResNet-50 image encoder — learned to recognise ImageNet classes three times slower than a much simpler baseline that predicts a bag-of-words encoding of the same text (§2.3).

The reason is the task. Both are trying to predict the exact words that accompany an image, and given the variety of descriptions, comments and related text that co-occur with images, that is simply too hard. So the paper switches to an easier proxy: predict only which text as a whole is paired with which image, not the words. Swapping the same baseline's predictive objective for a contrastive one bought a further 4× improvement in the rate of zero-shot transfer (§2.3).

The dot product becomes the distance

The mechanism is plain. Given a batch of $N$ (image, text) pairs, predict which of the $N\times N$ possible pairings actually occurred. An image encoder and a text encoder are trained jointly to maximise the cosine similarity of the $N$ real pairs and minimise it for the $N^{2}-N$ incorrect ones, optimising a symmetric cross-entropy loss over those similarity scores (§2.3). The embeddings come from a single linear projection of each encoder's representation — no non-linear projection.

Once training is done, images and sentences are unit vectors in the same space, and proximity is one dot product away. That is what "one map" means. The actual batch size is a very large 32,768, and the temperature $\tau$ is optimised directly during training (§2.5).