Paper Walkthrough: TLive-Omni — An Omni-Modal Model That Watches, Listens, and Tells You the Timestamp

A from-scratch walkthrough of TLive-Omni, an omni-modal understanding model built for e-commerce live streaming: Per-vGrid timestamped audio-video interleaving, a three-stage SFT recipe, and Faithful-RFT, which rewards answer faithfulness instead of visible reasoning — grounded strictly in the paper's own numbers.

TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Finding "that one sentence" inside a three-hour stream

Picture a live-commerce broadcast — a show where the host sells products while streaming. The host talks for three hours straight, products come and go on the desk, coupon text flashes in a corner of the frame, and viewers type "what sizes do you have?" into the chat. Now someone asks you: at what minute and second did the host mention the stock level for size L?

To answer, you need the speech (what was said), the video (what was in their hands at that moment), the on-screen text (prices and coupons), the product images (the catalogue's ground truth), and the viewer's question, all at once. And the supporting evidence sits somewhere in three hours of footage.

TLive-Omni, from the Taobao & Tmall Group of Alibaba, is an omni-modal understanding model built for exactly this situation. "Omni-modal" here means mapping image, video, audio, and text into a single shared representation space and treating them as one input sequence. The paper is explicit that this is a text-only-output understanding model (§2.1) — not a chatty companion, but a machine that reads the scene and answers questions about it.

The paper names two difficulties (§1). The first is interpreting heterogeneous signals jointly: process audio and video separately and merge the outputs, and you lose the fact that this utterance belongs to this moment of footage. The second is length — the evidence may be tens of minutes away. TLive-Omni supports up to 256K tokens of multimodal context (§1).

The obvious shortcut is to run an external ASR (automatic speech recognition) system, then feed only the resulting text to the model. The paper rejects this explicitly (§2.3). The moment speech becomes a transcript, the temporal correspondence between audio and video disappears, and so do paralinguistic cues such as who is speaking. In live commerce many product facts never appear in the frame at all and exist only in the host's voice — which is why the paper keeps audio as a first-class input modality.

Start with the scoring rubric: transcription is measured by edit distance

One of this model's main battlegrounds is transcribing stream audio. The paper scores it with CER (character error rate). Both the reference and the prediction are first normalized — speaker markers, bracketed tags, punctuation, spaces, and modal particles are stripped, and Chinese text is converted to simplified characters — and then (Appendix B):

In words, the score is a repair bill: a CER of 0.05 means five characters in every hundred still need fixing. Because the divisor grows with the reference, a host who talks for three hours is not penalized simply for having produced more text.

$S$ is substitutions (characters written as the wrong character), $D$ is deletions (spoken but missing from the transcript), $I$ is insertions (written but never spoken), and $N$ is the number of characters in the reference. In plain terms: how many characters would you have to change to repair the transcript, divided by how long the correct transcript is. That "how many characters to change" quantity is the edit distance, computed by filling a dynamic-programming table one cell at a time. Play with the figure below to see how the table fills in.

On-screen text recognition (OCR) is scored with the same instrument. The paper uses normalized edit distance, $d_{\mathrm{edit}}/\max(|p|,|g|)$, dividing by the longer of the predicted text $p$ and the ground truth $g$ so the score lands between 0 and 1 (Appendix B). Knowing the rubric first keeps the numbers later in the article from turning into noise.

Per-vGrid: putting the picture and the sound next to each other

Now the core idea. Per-vGrid lays out a video and its audio as a sequence of timestamped video grids (§2.4). The visual content of a time interval and the audio covering that same interval go into one span, wrapped in explicit boundary tokens, with a textual timestamp written at the front. It is less like keeping subtitles in a separate file and more like pasting each frame directly next to the line of script it belongs to. Compared with Qwen3-Omni, the paper lists four differences: an explicit prepended timestamp, explicit grid boundaries, keeping each grid's video and audio tokens contiguous, and separating neighbouring grids at the sequence level.

Why does mere adjacency help? A Transformer's input is, in the end, one flat token sequence, and the model uses positional proximity as one of its cues for relating things. If you place all the video tokens first and all the audio tokens after, two pieces of evidence about the same instant can end up thousands of tokens apart. Per-vGrid guarantees that corresponding evidence is adjacent and marks, with boundary tokens, where one moment ends and the next begins. The design point is that the correspondence is shown in the shape of the input rather than left for the model to infer.

The textual timestamp at the head of each grid is not decoration either. In temporal grounding — answering when something happened — the model ultimately emits seconds as text. If the seconds are already written in the input, the number to output can simply be copied from somewhere in the context. Which means that if the timestamps you wrote into the input drift from the actual footage, the model gets it wrong even when it copies correctly. That is why how you compute the timestamps matters.