Paper Walkthrough: Motion-Omni — Speaking and Moving in One Forward Pass

A walkthrough of Motion-Omni, a framework where a spoken dialogue model emits its spoken reply and its full-body gestures in the same pass. We follow the design that conditions motion on hidden states instead of waveforms, what it cost to train, and what the paper admits it has not shown.

Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Deciding what to say and how to move at the same time

When somebody says "there are three points here," three fingers usually go up. The voice and the gesture were not planned separately; they fall out of the same intention to speak. On the machine side, though, these two have lived in different model families for years.

The paper we are reading tries to fold that split into a single model. The original title is "Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue" (arXiv:2609.04250, submitted 28 August 2026), by Chengqian Ma, Wei Tao, Haoyu Zhang and Yiwen Guo.

Here is the abstract in plain terms. Spoken dialogue models (SDMs) produce a spoken reply but no motion; co-speech motion models produce motion from audio handed to them but never decide what to say. If you want both, the standard remedy is to run them in sequence — a cascade — which means a second full inference pass after the audio is finished, and which structurally forbids any motion objective from ever updating the speech or dialogue parameters. Motion-Omni is an end-to-end framework in which facial expression plus hand, upper-body and lower-body motion are emitted directly from the hidden states that produce the speech. The paper is blunt that joint training is not optional: with the speech pathway frozen, motion stays misaligned with the audio, and only co-adapting the LLM, Speech Generator and Motion Generator under both objectives recovers alignment while retaining spoken-dialogue ability. Supervision comes from a teacher pseudo-labeling pipeline yielding 422,856 quality-ranked pairs (1,402 hours), and the authors release SwDA-500 with an evaluation protocol. Built on a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 comes within 2% of the same-audio teacher cascade on reference-free motion metrics while responding 5.4× faster (RTF = 0.78, faster than real time), and reaches a 2.62% word error rate.

What a cascade is actually paying for

Two structural costs, per §1. The first is time: the motion model runs as a second full pass only after the audio is complete, so the wait stacks up. The second is learning: the motion loss never reaches the parameters that handle speech and dialogue. Any possibility of the speech side drifting toward "a way of talking that is easier to gesture over" is closed off from the start.

The paper asks whether both can be removed at once — whether motion can be a native output of an SDM, grown from the same states that produce the speech.

The shape of it: four components and where the conditioning points

Motion-Omni specifies four components (§3.1). The framework fixes only the input/output interfaces and the conditioning topology; the concrete model classes are choices made by a particular instance. Here is what the reference instance, Motion-Omni-Q7, picks.

• Speech Encoder / Projector: a frozen Whisper-large-v3 (hidden dim 1280). The projector concatenates every five consecutive frames, passes them through a two-layer MLP, and lands them in the LLM embedding space — a fivefold downsample of the sequence.
• LLM Backbone: Qwen2.5-7B-Instruct. Frozen through Stages 1–3, fine-tuned with a small learning rate only in Stage 4.
• Speech Generator: a Qwen2-style transformer initialised from Qwen2.5-0.5B-Instruct. It autoregressively emits GLM-4-Voice discrete speech units at 12.5 Hz over a vocabulary of 16,384 units plus three control tokens.
• Motion Generator: four parallel per-part decoders — face, hands, upper body, lower body — emitting LOM VQ codes at 30 Hz.

At inference, speech units go through a CosyVoice chunk-aware flow-matching decoder to mel spectrograms and then a HiFi-GAN-style vocoder to 22.05 kHz audio. Motion codes go through the frozen LOM VQ-VAE back into SMPL-X body/hand parameters and FLAME expression coefficients.

The mechanism: driving motion from hidden states, not waveforms

The single most consequential choice in this paper is what the Motion Generator conditions on. Not the decoded waveform, but the Speech Generator's last-layer hidden states (§3.2). Two input streams:

In words: $\mathbf{H}_s$, the Speech Generator's hidden states, projected into the motion working dimension, becomes the thing being looked at (keys and values). The speech units $\mathbf{u}$ just emitted, looked up in an embedding table $\mathbf{E}$, become the thing doing the asking (queries). $\mathrm{Interp}$ is linear interpolation along the time axis, stretching the 12.5 Hz speech-unit rate up to the 30 Hz motion rate. That interpolation is where the awkward fact that the two outputs run at different speeds gets absorbed.

The asymmetry carries meaning too. The paper's reading: the query side says "what is being said at this instant," one vector per speech unit, while the keys and values additionally carry things like speaker timbre — information that does not drive body movement. So something has to decide what gets through.

Token-as-Query Gated Fusion is that gatekeeper

Plain cross-attention would push the attention output straight into the residual stream. TQGF multiplies it element-wise by a sigmoid gate first (Appendix A.1). Each query token decides, coordinate by coordinate, how much of that signal to admit. And because the gated branch is added to the query rather than replacing it, a fully closed gate falls back on the un-contextualised query instead of producing a zero-information state.

The fusion ablation (Appendix G.1, BC under the Stage 3a setting) reads: TQGF with Speech Generator K/V is best at 0.912, plain cross-attention 0.905, FiLM 0.852, TQGF with LLM K/V 0.827, concatenation 0.809, GLU 0.799. The paper draws exactly two scoped conclusions from this — for the same TQGF operator, Speech Generator context beats direct LLM context; and gated fusion is modestly better than plain cross-attention in this fixed setup. It explicitly notes the experiment does not isolate every internal component of TQGF.