Paper Walkthrough: TLive-Omni — An Omni-Modal Model That Watches, Listens, and Tells You the Timestamp

A from-scratch walkthrough of TLive-Omni, an omni-modal understanding model built for e-commerce live streaming: Per-vGrid timestamped audio-video interleaving, a three-stage SFT recipe, and Faithful-RFT, which rewards answer faithfulness instead of visible reasoning — grounded strictly in the paper's own numbers.

TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Finding "that one sentence" inside a three-hour stream

Picture a live-commerce broadcast — a show where the host sells products while streaming. The host talks for three hours straight, products come and go on the desk, coupon text flashes in a corner of the frame, and viewers type "what sizes do you have?" into the chat. Now someone asks you: at what minute and second did the host mention the stock level for size L?

To answer, you need the speech (what was said), the video (what was in their hands at that moment), the on-screen text (prices and coupons), the product images (the catalogue's ground truth), and the viewer's question, all at once. And the supporting evidence sits somewhere in three hours of footage.

TLive-Omni, from the Taobao & Tmall Group of Alibaba, is an omni-modal understanding model built for exactly this situation. "Omni-modal" here means mapping image, video, audio, and text into a single shared representation space and treating them as one input sequence. The paper is explicit that this is a text-only-output understanding model (§2.1) — not a chatty companion, but a machine that reads the scene and answers questions about it.

The paper names two difficulties (§1). The first is interpreting heterogeneous signals jointly: process audio and video separately and merge the outputs, and you lose the fact that this utterance belongs to this moment of footage. The second is length — the evidence may be tens of minutes away. TLive-Omni supports up to 256K tokens of multimodal context (§1).

The obvious shortcut is to run an external ASR (automatic speech recognition) system, then feed only the resulting text to the model. The paper rejects this explicitly (§2.3). The moment speech becomes a transcript, the temporal correspondence between audio and video disappears, and so do paralinguistic cues such as who is speaking. In live commerce many product facts never appear in the frame at all and exist only in the host's voice — which is why the paper keeps audio as a first-class input modality.

Start with the scoring rubric: transcription is measured by edit distance

One of this model's main battlegrounds is transcribing stream audio. The paper scores it with CER (character error rate). Both the reference and the prediction are first normalized — speaker markers, bracketed tags, punctuation, spaces, and modal particles are stripped, and Chinese text is converted to simplified characters — and then (Appendix B):

In words, the score is a repair bill: a CER of 0.05 means five characters in every hundred still need fixing. Because the divisor grows with the reference, a host who talks for three hours is not penalized simply for having produced more text.

$S$ is substitutions (characters written as the wrong character), $D$ is deletions (spoken but missing from the transcript), $I$ is insertions (written but never spoken), and $N$ is the number of characters in the reference. In plain terms: how many characters would you have to change to repair the transcript, divided by how long the correct transcript is. That "how many characters to change" quantity is the edit distance, computed by filling a dynamic-programming table one cell at a time. Play with the figure below to see how the table fills in.

On-screen text recognition (OCR) is scored with the same instrument. The paper uses normalized edit distance, $d_{\mathrm{edit}}/\max(|p|,|g|)$, dividing by the longer of the predicted text $p$ and the ground truth $g$ so the score lands between 0 and 1 (Appendix B). Knowing the rubric first keeps the numbers later in the article from turning into noise.

Per-vGrid: putting the picture and the sound next to each other

Now the core idea. Per-vGrid lays out a video and its audio as a sequence of timestamped video grids (§2.4). The visual content of a time interval and the audio covering that same interval go into one span, wrapped in explicit boundary tokens, with a textual timestamp written at the front. It is less like keeping subtitles in a separate file and more like pasting each frame directly next to the line of script it belongs to. Compared with Qwen3-Omni, the paper lists four differences: an explicit prepended timestamp, explicit grid boundaries, keeping each grid's video and audio tokens contiguous, and separating neighbouring grids at the sequence level.

Why does mere adjacency help? A Transformer's input is, in the end, one flat token sequence, and the model uses positional proximity as one of its cues for relating things. If you place all the video tokens first and all the audio tokens after, two pieces of evidence about the same instant can end up thousands of tokens apart. Per-vGrid guarantees that corresponding evidence is adjacent and marks, with boundary tokens, where one moment ends and the next begins. The design point is that the correspondence is shown in the shape of the input rather than left for the model to infer.

The textual timestamp at the head of each grid is not decoration either. In temporal grounding — answering when something happened — the model ultimately emits seconds as text. If the seconds are already written in the input, the number to output can simply be copied from somewhere in the context. Which means that if the timestamps you wrote into the input drift from the actual footage, the model gets it wrong even when it copies correctly. That is why how you compute the timestamps matters.

The catch: it's the realized FPS, not the requested FPS

Per-vGrid derives each grid's timestamp and audio span from the frame indices that were actually sampled (§2.4), and integer rounding is where this bites.

Take a 119-frame source video at 30 FPS — roughly 3.97 seconds — and request 2 FPS sampling. The frames you actually get are $[0,20,39,59,79,98,118]$: seven of them. The realized sampling rate is $7/119\times 30 \approx 1.76$ FPS, not 2. Temporal patching groups two sampled frames into one grid (padding the final frame when needed), so seven frames form four grids, and a full grid spans about $2/1.76 \approx 1.13$ seconds rather than the 1.0 second the requested rate implies. At roughly 13 audio tokens per second, the audio span of a full grid becomes about 14–15 tokens instead of 13. Assign seconds using the requested rate and that discrepancy accumulates as timestamp drift.

The audio encoder and the three-stage recipe

The backbone is Qwen3.5, with the AuT audio encoder from Qwen3-Omni grafted on through a lightweight audio aligner (§2.1). AuT was trained from scratch on 20 million hours of audio, consumes 128-dimensional mel-spectrogram features at 16 kHz, and compresses speech to roughly 13 tokens per second (§2.3) — the compression is what keeps long recordings inside the context budget.

Training is a three-stage SFT (supervised fine-tuning) recipe, and the interesting part is what each stage freezes (§3.2).

Stage 1 does nothing more than map acoustic representations into the language model's embedding space. Stage 2 adds audio captioning and audio QA, pushing the audio pathway past transcription into sound events, music, and speaker-related cues. Stage 3 merges in image, video, and text supervision and picks up product grounding, temporal grounding, and the rest. Each stage runs for a single epoch, with learning rates dropping an order of magnitude each time: $1\times10^{-4}$, $1\times10^{-5}$, $4\times10^{-6}$ (Appendix D.1).

Mixing modalities makes sequence lengths within a batch wildly uneven, which inflates padding and skews the workload across workers. The paper's answer is a synchronized length-grouped sampler: partition by modality, sort each partition by token length, and have every worker use the same epoch-dependent seed so they all pull the same global batch at every step (§3.3).

Faithful-RFT: rewarding a faithful answer, not a visible thought process

The likelihood objective of SFT never incorporates task-specific feedback on what the model actually generated (§4). In live commerce, responses need to be faithful to the perceived evidence and fast enough to be useful. So the paper adds Faithful-RFT, running GRPO directly from the Stage-3 model with no separate cold-start SFT stage (§4.1). Its defining choice is that reasoning traces are not a reward target: no reward for reasoning length or visible chain-of-thought, and a lightweight format constraint that suppresses unnecessary think tags (§4.2). Longer generations mean more latency in production.

For each prompt $x_i$, the policy samples $G$ candidate responses, and the advantage — how good a candidate was — is computed relative to its own group.

In words: what drives learning is never a candidate's absolute score, only where it stands among the siblings that attempted the same prompt. An easy prompt where every candidate scores well and a hard one where every candidate scores badly still yield signals of comparable strength.

$R_{i,g}$ is the scalar reward attached to a candidate, mean and std are taken over the $G$ rewards produced from the same prompt, and $\epsilon_s$ is a small constant that avoids division by zero. Read it as a z-score: how much better than average was this candidate among its siblings who attempted the same problem?

The trap is that if all $G$ rewards are identical, the numerator is zero and the learning signal vanishes. The paper computes reward variance per group and, for groups with near-zero variance, regenerates the candidates with adjusted generation settings or rewritten prompts (generation is served by vLLM, §4.1). The knob being turned is on the sampling side — temperature and friends.

Rewards are selected by task-conditioned routing (§4.2). Each reward function $f_j$ declares an applicable task set $\mathcal{T}_j$, is evaluated only when the example's task falls inside it, and otherwise returns an invalid sentinel that is dropped before aggregation.

In words: a criterion that doesn't apply to this example is removed from the marking scheme rather than scored as zero, and the surviving weights are rescaled so they still sum to one. That is what stops an OCR penalty from landing on an example with no on-screen text at all.

$v$ is a 0/1 flag for "may this reward be applied to this example", $r$ is each reward function's score, and $\widetilde{w}$ is the configured weight renormalized over only the valid rewards. Multiple-choice questions, visual grounding, and OCR are handled by rule-based rewards; open-ended responses go to an LLM judge.

import numpy as np
R = np.array([0.9, 0.9, 0.9, 0.9])          # G=4 rewards from one group
A = (R - R.mean()) / (R.std() + 1e-4)       # -> all zeros. No gradient.
if R.std() < 1e-3:                          # the paper's dynamic resampling
    R = regenerate_group(prompt)            # rebuild with adjusted settings

w     = np.array([1.0, 1.0, 0.5])           # configured reward weights
valid = np.array([1,   0,   1])             # applicable to this task?
w_n   = w * valid / (w * valid).sum()       # renormalize over valid rewards

The reported settings are $G=8$, $\epsilon_s=10^{-4}$, clipping bounds of $0.2$ (lower) and $0.28$ (upper), and a KL coefficient of $0.1$ (Appendix D.2).

How to read the results

Evaluation runs along two axes: an in-house live-commerce suite and public general benchmarks (§5). On the in-house suite (§5.1), live-commerce ASR CER is 6.46 for the 9B model and 6.66 for the 4B, both below Qwen3-Omni's (30B-A3B) 6.75. Speaker-attributed cpWER is 12.27 for the 9B against Qwen3-Omni's 27.84 — a wide gap (Table 1). Product-image grounding AP reaches 91.45 for the 4B, though on live-stream frames Gemini 3.5 Flash leads at 84.15 (the 4B scores 82.85, Table 2). On video, the 9B reaches 81.49 temporal-grounding mIoU with a dense-caption hallucination rate of 8.76 (Table 3).

The general benchmarks (§5.2) are not a clean sweep. In Table 4 the 4B scores 70.9 on MMMU against its own Qwen3.5 4B backbone's 72.1, and 79.9 versus 81.0 on MathVista — while beating the backbone on DynaMath (72.5 vs 69.6) and RealWorldQA (77.7 vs 72.5). On TimeLens-Bench (Table 7) the 4B outscores the 9B on all three benchmarks, so bigger is not uniformly better here.

Appendix C's in-context ASR results are the most operationally useful. Given a candidate keyword list, the 9B goes from 48.62 recall / 6.54 CER with no list to 81.00 / 5.18 with 50 keywords, and CER stays at 5.53 even with a 1000-keyword list. The comparison model Qwen3-ASR-Flash degrades to 10.68 CER at 1000 keywords — not being dragged down by a long dictionary is the differentiator (Table 9).

The limitations the paper states honestly

The paper names three (§6). TLive-Omni is scoped to understanding, not generation or full-duplex real-time interaction. Evaluation coverage on broader public benchmarks should be widened. And robustness to longer, noisier, more varied streams — plus calibration of temporal evidence under incomplete or ambiguous inputs — needs strengthening.

Three more caveats worth adding as a reader. First, the evaluation suite is built in-house (§5.1), so nobody outside can reproduce those numbers under identical conditions. Second, the accuracy and hallucination figures for audio description and dense captioning come from a separate LLM judge that answers multiple-choice questions using only the generated text (Appendix B) — in practice you should assume those numbers move if the judge changes. Third, there is no ablation of Faithful-RFT, so the paper does not let you attribute any share of the final performance to the RFT stage specifically.

How this shows up in practice

The people who touch this are multimodal platform teams at e-commerce or streaming companies, and the operations side running moderation and product-information extraction.

• Don't collapse audio to text up front. Passing only an external ASR string downstream throws away temporal correspondence and speaker cues (§2.3).
• Watch the realized FPS, not the requested FPS. Setting fps=2 does not give you 2 (in the paper's example it is 1.76). If you don't derive seconds from the frame indices you actually sampled, your mIoU quietly drops for no visible reason.
• A bigger in-context ASR keyword list is not automatically better. As Appendix C shows, some models get worse CER at 1000 keywords. In practice, sweep the list length and watch both CER and recall in an A/B before committing.
• Monitor per-group reward variance. If you run GRPO yourself, put the fraction of near-zero-variance groups on a dashboard. When it stays high, training is running but no gradient is forming.
• The design-review question you'll get: "how do you align audio with video?" Lay out the options — (1) encode separately and fuse, (2) interleave alternately, (3) put them adjacent inside timestamped grids — and argue for (3) when long-range evidence lookup is required.

For background, Why Whisper is robust, Video understanding from scratch, and Instruction Tuning and RLHF are the shortest routes.

Takeaway

Compressed to one line, the claim is that aligning model design, data construction, training objectives, and evaluation protocol with the target deployment scenario is what matters (§6). This is not a flashy new architecture — it is a paper built out of unglamorous consistency: how timestamps are computed, what gets frozen in which order, and where the reward is pointed.