Paper Explained: Ovis-Embedding — One Ruler for Text, Images, Video, and Audio
A from-scratch walkthrough of Ovis-Embedding, a model family that puts text, images, video, and audio into one vector space: the four-stage training recipe, the trick that keeps accuracy when you shorten the vectors, and what the paper does not show.
Key points
- Ovis-Embedding is a family of omni-modal embedding models that uses one shared backbone to map text, images, video, and audio into a common representation space.
- The three advances are native omni-modal initialization from a Qwen-omni model, data-centric training, and embedding-specific optimization.
- Homogeneous-source sampling builds each batch from a single source so the in-batch negatives are informative.
- Low-rank feature decomposition at inference time lets you shrink the vector dimension, and state-of-the-art results are reported on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB.
- Ovis-Embedding uses cosine similarity for both training and retrieval, i.e. the dot product of vectors scaled to length 1.
Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings
Primary source — what this article is built on
undefined2026-09-21→undefined2026-09-26same month
Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal EmbeddingsOvis-Embedding Team · 2026-09-21 · v1arXiv:2609.25165Paper page·PDFundefined
In this report, we introduce \textbf{Ovis-Embedding}, a state-of-the-art omni-modal embedding family built on native integration of text, image, video, and audio. Instead of assembling separate modality towers, Ovis-Embedding uses a shared multimodal backbone to encode different modalities in a common representation space. Specifically, we make \textbf{three key advances}: (1) \textbf{native omni-modal initialization}: we adopt a pretrained Qwen-omni model as the embedding backbone and adapt it through contrastive training with low-rank initialization; (2) \textbf{data-centric omni-modal training}: we construct a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To improve data efficiency, we introduce homogeneous-source sampling to form task-consistent batches with informative in-batch negatives; and (3) \textbf{embedding-specific training and inference optimization}: we use focal loss to emphasize hard examples and similarity-based Embedding Distillation to transfer fine-grained similarity structure from complementary experts. At inference time, low-rank feature decomposition enables compact embeddings with flexible dimensionality and minimal performance loss. Empirical evaluations show that the \textbf{Ovis-Embedding} family achieves state-of-the-art performance on \textbf{MMEB-v3}, \textbf{MMEB-v2}, \textbf{MVEB}, \textbf{MAEB}, and \textbf{RTEB}, demonstrating its effectiveness across text, image, video, and audio modalities. These results highlight the potential of unified omni-modal training to overcome modality fragmentation and advance universal embedding models for any-to-any retrieval.
What this paper does
This article covers the technical report "Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings" by the Ovis-Embedding Team at Alibaba (arXiv:2609.25165, published September 21, 2026).
Here is what the abstract says. Ovis-Embedding is a family of "omni-modal" embedding models built on a backbone that handles text, images, video, and audio natively. Instead of bolting together a separate encoder for each modality, it uses one shared backbone to map every kind of input into a common representation space. The authors describe three advances. (1) Native omni-modal initialization: they start from a pretrained Qwen-omni model and adapt it with contrastive training using low-rank (LoRA) initialization. (2) Data-centric training: a broad corpus of text, images, video, audio, and interleaved inputs, plus homogeneous-source sampling, which builds each batch from a single source so the in-batch negatives are informative. (3) Embedding-specific optimization: a focal loss that emphasizes hard examples, similarity-based embedding distillation from complementary expert models, and low-rank feature decomposition at inference time that lets you shrink the vector dimension with minimal loss. They report state-of-the-art results on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB.
In one line: it's a design and training recipe for measuring "how close" any two things are with the same ruler, whatever kind of data they are.
An analogy: one shelf-numbering system for the whole library
Picture a library that holds books, photos, film reels, and audio tapes side by side. Patrons ask for things across those types. The paper's introduction (§1) gives the example of a maintenance agent that takes a recording of an abnormal machine sound plus a short text description and needs to find a video tutorial, a diagram in a PDF manual, or an old service record.
The naive approach is to hire a separate librarian for each media type, each with their own numbering scheme. Nobody can then tell whether "Photo shelf 3" and "Audio shelf 3" mean the same thing. What you actually want is one librarian who numbers every item under a single shared rule. The paper calls this any-to-any retrieval, and argues that emitting vectors of the same length is not enough. You need a space where scores are comparable across modalities and the fine distinctions inside each modality survive (§1).
The authors' concern is that most existing omni-modal embedders take a space built from images and text and attach an audio pathway afterward. Audio then gets squeezed into a shelf layout that was designed without it, which may hurt fine-grained alignment (§1).
Intuition: an embedding is a coordinate for meaning
An embedding turns an input into a list of a few thousand numbers, a vector. Training pushes things with similar meaning toward vectors that point in similar directions. The basics are covered in Embeddings from Scratch.
Search then scores how closely the query vector points toward each candidate and ranks the candidates. Drag the query in the figure below and watch the top five change. Switch between dot product and cosine and you'll also see that normalizing lengths (or not) changes who makes the list. Ovis-Embedding uses cosine similarity for both training and retrieval, i.e. the dot product of vectors scaled to length 1 (§2).
Comments
Sign in to comment