Paper Deep Dive — CLIP: Putting Words and Images on One Map

A reading of the CLIP paper (Radford et al., 2021) grounded strictly in its own text: how the in-batch positives and negatives work, why zero-shot classification is really a hypernetwork trick, the five points that prompt wording is worth, and the long list of limits the authors wrote themselves.

Learning Transferable Visual Models From Natural Language Supervision

If it does not fit in one of a thousand boxes, it is invisible

State-of-the-art vision systems are trained to predict a fixed set of predetermined categories. That form of supervision limits their generality, because specifying any other visual concept means collecting more labelled data. That is where the paper opens (Abstract).

The weakly supervised work that came before sat in the same cage. The paper points out that those approaches carefully design — and in the process limit — their supervision to 1000 classes, or 18291 classes. They also predict through a static softmax classifier with no mechanism for dynamic outputs, which severely curtails flexibility and caps zero-shot ability (§1). Meanwhile the attempts to learn directly from natural language performed poorly: Visual N-Grams reached 11.5% zero-shot on ImageNet (§1).

So the authors collect 400 million (image, text) pairs from publicly available sources on the internet, searching for text containing one of 500,000 queries (words appearing at least 100 times in English Wikipedia, WordNet synsets and so on) and class-balancing with up to 20,000 pairs per query. They call it WIT, for WebImageText (§2.2).

Predicting the caption was the long way round

The first approach jointly trained an image CNN and a text transformer to predict the caption. Efficiency never came. Figure 2 shows the measurement: a 63-million-parameter Transformer language model — already using twice the compute of its ResNet-50 image encoder — learned to recognise ImageNet classes three times slower than a much simpler baseline that predicts a bag-of-words encoding of the same text (§2.3).

The reason is the task. Both are trying to predict the exact words that accompany an image, and given the variety of descriptions, comments and related text that co-occur with images, that is simply too hard. So the paper switches to an easier proxy: predict only which text as a whole is paired with which image, not the words. Swapping the same baseline's predictive objective for a contrastive one bought a further 4× improvement in the rate of zero-shot transfer (§2.3).

The dot product becomes the distance

The mechanism is plain. Given a batch of $N$ (image, text) pairs, predict which of the $N\times N$ possible pairings actually occurred. An image encoder and a text encoder are trained jointly to maximise the cosine similarity of the $N$ real pairs and minimise it for the $N^{2}-N$ incorrect ones, optimising a symmetric cross-entropy loss over those similarity scores (§2.3). The embeddings come from a single linear projection of each encoder's representation — no non-linear projection.

Once training is done, images and sentences are unit vectors in the same space, and proximity is one dot product away. That is what "one map" means. The actual batch size is a very large 32,768, and the temperature $\tau$ is optimised directly during training (§2.5).

What zero-shot classification actually is

Zero-shot classification is nothing more than reusing that pre-training capability. Take the names of every class in a dataset as the candidate texts, compute the cosine similarity against the image embedding, scale by the temperature, and normalise with a softmax (§3.1.2). Written out, with $\mathbf{v}$ denoting L2-normalised embeddings:

$\langle\cdot,\cdot\rangle$ is the inner product — which, on normalised vectors, is exactly the cosine similarity — and $\tau$ is the temperature. The paper describes this prediction layer as "a multinomial logistic regression classifier with L2-normalized inputs, L2-normalized weights, no bias, and temperature scaling" (§3.1.2). The temperature is learned, initialised to the equivalent of 0.07 and clipped so the logits are never scaled by more than 100 (§2.5).

Read this way, the text encoder is a hypernetwork that generates the weights of a linear classifier, and the image encoder is just a backbone (§3.1.2).

Wording the prompt is worth five points

Embedding bare class names creates two problems. One is polysemy: ImageNet contains both construction cranes and cranes that fly, and the boxer class in Oxford-IIIT Pets clearly means a dog breed in context, but to a context-free text encoder it could be an athlete. The other is a distribution gap — text in the pre-training data is usually a sentence, not a single word.

The fixes are concrete. Using A photo of a {label}. as the default alone improves ImageNet accuracy by 1.3%. For satellite imagery a satellite photo of a {label}. worked. Ensembling 80 different context prompts adds another 3.5%, for roughly 5% in total — comparable to using 4× more compute, and effectively free, since the ensemble is built in embedding space and amortises to the cost of a single classifier (§3.1.4).

The strengths and the failures are both sharp

Across the 27-dataset suite, zero-shot CLIP beats a supervised logistic regression fitted on ResNet-50 features on 16 of them. On ImageNet it reaches 76.2%, matching the original ResNet-50 without using a single one of the 1.28 million labelled examples, and on video action recognition it gains 14.5% on Kinetics700 and 7.7% on UCF101 (§3.1.3, §3.1.5).

The weak spots are equally clear: satellite imagery, lymph node tumour detection, counting objects in synthetic scenes, traffic signs, distance to the nearest car. Even on fine-grained tasks it loses by more than 10% on car models and aircraft variants. Measured as data efficiency, zero-shot is on par with a 4-shot logistic regression on the same feature space (§3.1.5).

Robustness deserves its own note. Across seven natural distribution shifts, ImageNet models lose a great deal of accuracy, and zero-shot CLIP closes up to 75% of that robustness gap. Yet adapting CLIP's features to ImageNet raises ImageNet accuracy by 9.2% to 85.4% while average accuracy under distribution shift slightly decreases. The paper says it does not have a confident answer for why (§3.3).

The limits the paper lists in §6

• Not enough compute. Reaching overall state of the art zero-shot is estimated to need around 1000× more compute, which is infeasible on current hardware.
• Truly out-of-distribution data breaks it. CLIP scores only 88% on MNIST handwritten digits, losing to logistic regression on raw pixels. The authors themselves call the hope that a large and varied dataset makes everything effectively in-distribution "a naive assumption that, as MNIST demonstrates, is easy to violate".
• It can only pick from the classes you give it. Nothing like the open-ended output of a captioning model.
• Methodological caveats. Full validation sets were queried repeatedly during development, and the 27-dataset evaluation suite is admitted to be "undeniably co-adapted with the development and capabilities of CLIP".
• Social bias. In §7.1, adding non-human classes to 10,000 FairFace images misclassified 4.9% of them, with images labelled "Black" at roughly 14% against under 8% for others; crime-related misclassification ran 16.5% for male images versus 9.8% for female.
• Zero-shot beats one-shot, counter-intuitively. Humans go from 54% to 76% with a single example (§4); CLIP gets worse the moment you switch to a linear classifier.

Note as of writing (2026) — an author's remark, not a claim of the paper: every number is this paper's own from 2021, with no later reproductions or improved variants included.

How this shows up on the job

CLIP shows up in production as similarity search for image retrieval and recommendation, as a moderation filter, and above all as a way to sift unlabelled image piles using words. In most of those cases nothing is trained.

The knobs are the prompt and the threshold. Do not feed bare class names; start from A photo of a {label}. and add domain words. Averaging several templates in embedding space and caching the result raises accuracy without raising cost (§3.1.4). The temperature is a learned value you simply use as-is.

Ways to get hurt. (1) Forget the L2 normalisation and the dot product stops being a cosine, so your threshold means nothing. (2) Polysemous class names — crane and boxer are the paper's own examples. (3) Counting and measuring barely work. (4) Nothing outside the class set can ever come out, so if you need "none of the above", write that sentence into the candidate set yourself (§3.1.5, §6).

Questions that come up in interviews. "Walk through zero-shot classification." — the text side builds a weight vector per class, take the dot product with the L2-normalised image embedding, divide by the temperature, softmax (§3.1.2). "Why contrastive rather than captioning?" — predicting exact words is too hard, and Figure 2 measured a 3× plus 4× efficiency gap (§2.3). "What does batch size affect?" — negatives come from the $N^{2}-N$ mismatched pairs inside the batch, so the batch determines how many negatives you get. Note, though, that the paper never tests a smaller batch. For the image-side backbone, see the ViT paper deep dive.

Summary

• CLIP is contrastive learning over a single question: which image goes with which sentence in this batch. $N$ positives, $N^{2}-N$ negatives, symmetric cross-entropy (§2.3)
• Zero-shot classification is best understood as the text encoder generating the weights of a linear classifier (§3.1.2)
• Prompt design and ensembling move ImageNet accuracy by about 5% (§3.1.4)
• It is robust to distribution shift, but adapting it to ImageNet erases most of that advantage (§3.3)
• 1000× compute to reach SOTA, 88% on MNIST, nothing outside the class set — the limits are the paper's own (§6)

Original paper: Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (arXiv:2103.00020, 2021)