Paper Explained: Realtime-Venus — A Voice-and-Video Assistant That Keeps Talking While Work Runs in the Background

Listening, speaking, and background delegation all share one clock ticking once per second. We walk through the 9B models and the harness that let an assistant hand off tool calls without pausing the conversation, from analogy to equations to pseudocode.

Realtime-Venus: A full-duplex interaction system with asynchronous delegation

A receptionist who passes notes while staying on the phone

This article covers "Realtime-Venus: A full-duplex interaction system with asynchronous delegation" (arXiv:2609.13814) by the Venus Team at Ant Group, with Tsinghua University.

Here is what the abstract claims. Natural interaction needs continuous perception and timely responses, and video interaction additionally needs the conversation grounded in a visual scene that keeps changing. The authors present a proactive full-duplex system built from two separately trained 9B models: Realtime-Venus-Omni for audio–visual interaction and Realtime-Venus-Audio for spoken interaction. Each model is a complete conversational frontend that combines perception, conversational control, and native speech generation, placing user inputs, model outputs, and delegation events on one shared causal timeline. A dual-loop runtime lets the live conversation continue while Realtime-Venus-Harness executes tasks asynchronously and feeds the results back into the dialogue. Among the online models evaluated, the Omni model scores highest on six of eight video benchmarks, and on Full-Duplex-Bench v1.5 the Audio model keeps talking through backchannels, speech aimed at someone else, and background speech 97%, 88%, and 86% of the time (Abstract).

Think of a receptionist who never hangs up the phone but slides a note to a colleague asking them to look something up. When the caller says "uh-huh," the receptionist keeps going. When the caller says "actually, make that Thursday," only the part not yet spoken gets rewritten. When the colleague comes back with an answer, the receptionist relays it at a natural pause. Older voice assistants work more like walkie-talkies: talk when the other side goes quiet, stop the moment they speak. This paper replaces that with a design where listening, speaking, delegating, and receiving results all run at once on the same clock.

Intuition: three streams woven together every second

The conversation is sliced into one-second chunks, and each chunk aligns three streams (§4.3):

• User stream: visual and audio features. It keeps flowing while the model is speaking
• Assistant stream: displayed text, speech tokens, and a hidden "delegation request"
• Background stream: results from background work. A task that hasn't finished contributes nothing, so the foreground never waits

Every second, the model picks "listen" <|listen|> or "speak" <|speak|>, and closes its turn with <|turn_eos|> (§4.1). To hand work off, it writes plain language inside <delegate>…</delegate>, which is never shown or spoken. The result later arrives on the input side as <backend>…</backend> (§4.3).

Because these decisions come from the same semantic representation that writes the reply, overlapping audio is not treated as a blanket "stop" signal (§4.4). A backchannel means keep talking. A correction or a new request means cut off the stale answer and regenerate only the unspoken continuation, leaving already-played audio alone. The paper splits interruption handling into three actions — stop and discard, repair with new information, or park the current plan and follow the new request — and trains the choice together with what gets said next. That is how acoustically similar overlaps can lead to different continuations, which is the key difference from binary reactions driven by voice activity detection (VAD).