Self-Supervised Learning — The Day Unlabeled Data Became an Asset

Nobody has to label the data — the data can write its own exam. A ground-up tour of the two families (masked prediction and contrastive learning), from the intuition through the InfoNCE loss, an interactive figure, PyTorch code, and finally why LLM pretraining is the largest self-supervised system ever run.

A Simple Framework for Contrastive Learning of Visual Representations

There Are Not Enough People to Label Things

Most of the progress in image recognition through the 2010s sat on top of ImageNet: a dataset where humans hand-named photographs on the order of fourteen million times. A cat photo gets "cat", a ship gets "ship". Only after that patient work can a model learn the mapping from input to answer.

That approach has a hard ceiling, though. The rate at which you can produce labels becomes the ceiling on how much data you can train on. Servers around the world hold billions of images and trillions of words, and almost none of it carries a name. In domains like medical imaging, where only a specialist can supply the label, the wall is higher still — and money doesn't knock it down, because the expert's time is the finite resource.

Self-supervised learning (SSL) doesn't attack that wall so much as walk around it. Instead of asking a person for the answer, you hide part of the data and let the machine set itself the problem of recovering it. The answer was inside the original data all along, so the human cost is zero.

The idea is old. What changed around 2020 is that representations built without labels started matching representations trained with them, one result after another, until the ordering flipped: build the foundation on raw data first, and spend your scarce labels last. Data that had been dead weight in a warehouse turned into an asset overnight.

Setting Your Own Exam

Concretely, you are writing your own fill-in-the-blank drills.

Delete one word from "Today I went to ___" and the answer is the word you deleted. Cover the right half of a photo and the answer is that right half. Take two different crops of one photo and you have the question "did these two come from the same original?" No human touched any of it.

A task built this way is called a pretext task — a pretext, not a goal. You don't actually want a model that is good at filling blanks. You want the representation it has to build in order to fill them: the ability to turn an input into a vector where similar meanings land near each other.

The analogy is strength training versus sport. Nobody wants to be good at squats for their own sake, but the legs you build carry over to football and basketball alike. The pretext task is the squat; the classification or search problem you actually care about is the game. That two-stage shape — pretraining for general strength, then a small amount of labeled data to specialize (fine-tuning) — is now the default way AI systems get built.

Which means the design of the pretext task decides everything. Pick a problem that's too easy and the model scores full marks by copying nearby colors without understanding anything. How to build a task that is hard enough, yet only solvable by understanding the content — the history of self-supervised learning is largely the history of that one question.

Two Families

Depending on how you pose the problem, SSL splits into two lineages.

Predictive (generative) methods hide part of the input and ask the model to reconstruct exactly what was hidden. BERT's masked words, GPT's next token, MAE for images. The premise: if you can reconstruct it, you must have understood it.

Contrastive methods never reconstruct anything. They only ask which items belong together. Two views of the same image should land close in vector space; anything from a different image should land far away. SimCLR, MoCo, and CLIP are this lineage.

Both share the property that no human writes an answer key. What differs is how fine-grained the grading is. Predictive methods grade down to individual pixels and tokens; contrastive methods only ask a coarse same-or-different question. Finer grading extracts more signal, but it also spends capacity dutifully modeling details that carry no meaning — sensor noise, paper texture.

The core of a contrastive method is genuinely simple. Measure closeness between two vectors with a dot product (cosine similarity once you normalize), then push it up for partners and down for strangers. That's the whole mechanism.

The Contrastive Recipe in Four Lines

The SimCLR procedure (Chen et al., 2020) fits in four steps.

1. Apply a random transform twice to one image $x$, producing two views $\tilde{x}_i$ and $\tilde{x}_j$
2. Run them through an encoder $f$ (a ResNet, say) to get features $h$
3. Push those through a small MLP (the projection head) to get $z$
4. Within the batch, pull $z_i$ and $z_j$ together and push every $z$ from other images away

The thing playing the role of teacher here is data augmentation. You are encoding a human intuition — "different crop, different tint, still the same object" — as a list of transforms. Which means your choice of augmentations literally is your definition of what counts as the same thing.