Paper Walkthrough: WeMM-Embedding — Putting Text, Images and Video on One Ruler

A walkthrough of WeMM-Embedding (2B/4B/9B), Tencent's universal multimodal embedding family. The unified pair format, the <embedding> token, Matryoshka dimensions, two-stage training and distillation — explained from scratch, using only the numbers the paper reports.

WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report

The job: turning meaning into an arrow

Walk into a bookshop and say "that novel I read last year, red cover, there's a cat in it," and a good bookseller walks you to the shelf without ever hearing the title. The thing doing that job inside a search or recommendation system is an embedding model. It turns content — a paragraph, a photo, a clip — into a list of a few hundred to a few thousand numbers, arranged so that things which mean similar things point in similar directions. Once you have that, measuring the angle between two arrows is enough to search, rank, and classify. How such a space gets built is covered in Embeddings from Scratch; here the question is narrower: what gets to live in the space.

The awkward part is that real catalogues aren't one thing. Articles, thumbnails, short videos, screenshots of PDFs — and queries like "something like this photo, but in red," where an image and a sentence arrive together. CLIP-style models keep separate encoders for images and text, so they never had a natural way to fold interleaved input into a single vector (§1).

The paper here is WeMM-Embedding, a technical report published in August 2026 by the WeChat Vision team at Tencent. It's a family of models (2B, 4B, 9B) that maps text, images, video, visual documents, and any interleaved mixture of them into one shared space, with weights and code released (§1).

The claims, up front

• On MMEB-v2 (78 datasets), the 2B model scores 77.9 overall — edging past Qwen3-VL-Embedding-8B at 77.8, the strongest open-source baseline of the previous generation (§4.1.1)
• The 9B model reaches 80.6, first on the official leaderboard as of August 24, 2026 (§1)
• On a 26-task in-house benchmark drawn from real WeChat products, the 2B model scores 72.0 against 60.9 for Qwen3-VL-Embedding-2B, plus improvements in 14 online A/B tests (§4.3)

The framing the authors choose is a moved performance–efficiency frontier: a small model overtaking a larger one from the previous generation (§1). Let's work through it in order — data, model, training, evidence.

Forcing everything into one shape (§2.1)

The design decision doing the most work here is an unglamorous one: every kind of task gets rewritten into the same single row. The paper represents each training example as a five-tuple.

Reading the symbols in order: $I_i$ is an optional instruction stating what kind of match is wanted; $q_i$ is the source, the thing doing the looking; $c_i$ is the target that should be found; $\mathcal{N}_i$ is an optional set of hard negatives, candidates that are close but wrong; $y_i$ is an optional graded relevance score. Both $q_i$ and $c_i$ may be text, an image, a video, or a mixture.

Put plainly: the searcher, the thing searched for, the tempting near-misses, and how close they are. That's enough to express captioning, classification, QA, retrieval and recommendation data in one notation. Classification becomes "image → class name (or a description of the class)"; QA becomes "question → answer" (§2.1). Once the shape is uniform, everything flows through a single pipeline — and the targets belonging to other examples in the same batch become negatives for free (in-batch negatives). That detail comes back to bite later.

It's worth pausing on $I_i$. Two examples can have identical modality structure and still want completely different matches: "photos of this same product" and "a sentence describing this image" both start from an image, but the right answers have nothing to do with each other. The instruction is the tag that tells the model which relation is in play, and removing it costs measurable accuracy later (§4.4.2). Operationally, an instruction is less like a prompt and more like a declaration of what relation an index was built under.

Curating the data again (§2.2)

Stage 1 trains on several hundred million pairs, but the authors also build a curated set roughly one tenth that size. The goal isn't volume; it's flattening the semantic skew and making the supervision more informative. Three steps (§2.2).

First, Semantic-ID-guided resampling. For each pair, whichever side has the longer token sequence is encoded with an intermediate WeMM-Embedding checkpoint, and a three-level residual k-means quantizer (RQ-KMeans) assigns it a three-element discrete ID. Codes with many examples crowded into them get sampled down; sparsely populated codes get kept at higher rates. Notably they don't force a uniform distribution — they only shave off repeated exposure to frequent patterns. Second, quality refinement: a multimodal LLM checks whether each pair actually reflects the intended relation and drops the ones that don't, and rewrites noisy text — for web alt-text, correcting factual errors while preserving the original style and level of detail. Third, hard-negative construction: for text targets, an LLM writes plausible-but-wrong candidates; for image and video targets, intermediate checkpoints of the model itself retrieve lookalikes from task-specific pools. The theme isn't more data — it's sharper objections from the teacher.

The model: an <embedding> token at the end (§3.1)

The models are built on the natively multimodal Qwen3.5 backbones (§3). Text goes through the tokenizer, images and video through the native visual pipeline, and the resulting tokens are laid out in the order the input presented them. A dedicated <embedding> token is appended at the end, and its final-layer hidden state, L2-normalized, is the output vector — last-token pooling.

Because attention is causal, that final token has already seen everything before it. The application the paper gives makes the value concrete: when a video is followed by its ASR transcript, you can place one <embedding> token right after the video tokens and another at the very end, and a single forward pass yields both a video-only representation and a joint video-plus-text one (§3.1).