How VLMs Came Together — Wiring a Vision Encoder into an LLM

A vision-language model is three parts: a vision encoder, a projection layer, and an LLM. From the bridge CLIP built, through the three schools of connector design, to resolution strategies and the implementation constraints that bite in practice.

Learning Transferable Visual Models From Natural Language Supervision

How did language models learn to "read" images?

Hand ChatGPT a photo and it will describe what's in it. Yet the LLM at its core is a machine that only processes sequences of text tokens. Image support was not achieved by rebuilding the LLM from scratch. Instead, engineers bolted on a separate part that plays the role of eyes, and converted its output into a format the LLM can read. The whole assembly is called a VLM — a Vision-Language Model.

This article explains, assuming no prior knowledge, what parts a VLM is made of, what the "projection layer" that glues them together actually does, and why the current battleground is resolution strategy.

An analogy: the foreign correspondent and the interpreter

A VLM works like a news broadcast with an interpreter. There are three people involved.

1. The vision encoder is a foreign correspondent. It goes to the scene (the image) and writes detailed notes (a sequence of vectors) — but the notes are written in the correspondent's native tongue, the "language of vision"
2. The projection layer (the projector) is the interpreter. It translates the correspondent's notes into words the anchor can read
3. The LLM is the anchor. It reads the translated notes together with the viewer's question (the text prompt), and speaks

The crucial point: the anchor never sees the scene. Only the correspondent did, and all the anchor receives is the translated notes. A VLM's quality is the product of three factors — how observant the correspondent is, how faithful the interpreter is, and how articulate the anchor is.

The intuition: turn the image into a "foreign-language sentence"

To an LLM, input means a sequence of embedding vectors. For text, each token becomes one vector. So if we can turn an image into a sequence of vectors too, the LLM should be able to read it without distinguishing it from text — that is the single idea underlying nearly every modern VLM.

The recipe:

1. Cut the image into small patches, e.g. 14×14 pixels (exactly as in a Vision Transformer)
2. The vision encoder turns each patch into a vector that reflects its surrounding context
3. The projection layer maps those vectors into the LLM's embedding dimension
4. Splice the resulting vector sequence directly into the sequence of text tokens

From the LLM's point of view, an image is "a sentence of a few hundred foreign words it has never seen." Through training, it learns to read that foreign language.

The founding moment: CLIP built the bridge

For years, the near-default choice of bolt-on eyes was the image encoder from CLIP, published in 2021. CLIP collected a huge set of image–caption pairs and trained an image encoder and a text encoder jointly, so that embeddings of matching pairs end up close together and mismatched pairs end up far apart (contrastive learning).

"Closeness" is measured with cosine similarity.

Here $f(I)$ is the embedding vector of image $I$, $g(T)$ is the embedding vector of text $T$, the $\cdot$ in the numerator is the dot product, and $\lVert \cdot \rVert$ is the length of a vector. In plain words, equation (1) just measures how small the angle between two vectors is — the more the image and the caption point in the same direction, the closer the value gets to 1.

As a by-product of this training, CLIP's image encoder learned to summarize images in a form that lines up naturally with language. That is precisely why later VLMs could simply borrow this encoder, frozen, and get away with it.