Paper Walkthrough: EchoWM — A World You Can Walk Into, and Hear, Driven by Camera Intent

Hand it a reference view and a stream of 'where I want to go', and it keeps generating 720p video together with environmental sound, music and speech. A walkthrough of EchoWM from the paper itself: one camera-intent interface for both first and third person, a single translation scale shared across the whole dataset, and the sink-plus-FIFO cache that makes long rollouts possible.

EchoWM: Open and Enterable Omnimodal World Models

The one-sentence version

EchoWM (arXiv:2608.23189) is a world model that takes a reference observation, a text description, and a continuous stream of navigation input, and keeps generating 720p video together with environmental sound, music and speech (Abstract). The defining move is that first-person and third-person scenes are driven by the same user-facing input: camera intent. It is a 42-page technical report from a joint team at HKUST, Peking University, JD's Joy Future Academy and others, with a project page and a code link listed on the paper.

A cinema, and a theme park

Start with the metaphor. Today's video generation is a cinema. You hand over a prompt like an order slip, take your seat, and watch whatever comes out. World models invert that relationship: they generate what happens next in response to continuous input from a user or an agent (§1). That is a theme park.

But, the paper continues, most existing world models either emit silent rollouts or expose a rigid action space tied to one particular game or controller (§1). And in an interactive world, audio is not decoration — it is information. Footsteps and collisions give physical feedback, ambient sound conveys spatial dynamics, and speech ties visible subjects to social and narrative context (§1).

Table 1 lays out the reported native capabilities of representative video and world models: high-resolution video, first- and third-person support, continuous trajectories, discrete keyboard control, environmental sound, background music, and spoken speech. EchoWM is the only entry with all seven marked (Table 1). Genie 3, for instance, is listed with discrete keyboard control but no audio; LTX-2.3 has all three audio capabilities but no participation interface at all. The capabilities exist — they are just scattered across different systems.

The authors call their target an enterable omnimodal world, and they fence it in from the start: interaction here means navigation and viewpoint evolution that can be written as a continuous trajectory, not unrestricted control over arbitrary subject behaviour (§1). That boundary stays consistent all the way through to the limitations section.

The difficulty is three kinds of mismatch

The paper names three coupled obstacles and assigns a design to each (§1).

The first is architectural: heterogeneous video sources differ fundamentally in metric scale and in control semantics, and naive normalization either erases displacement magnitude or degrades control consistency. The second is data: real-world video has rich acoustics but messy motion, while simulation has precise geometry, simple semantics, and no sound at all. The third is training: audio-visual quality and control responsiveness have their cleanest supervision in different subsets, and forcing them together induces distribution conflicts.

The three answers are, in order: a unified camera-intent interface, a four-source data engine, and a four-stage curriculum.

Camera intent as the single interface

Here is the core idea. Express user control not as a per-subject action vocabulary, but as how the observer intends to move and to look — camera intent (§1, §4.2.1).

In first-person scenes the trajectory directly drives observer ego-motion. In third-person scenes the same trajectory specifies camera evolution, while the associated character locomotion and camera-following behaviour are learned from data (§4.2.1). The model is given no explicit character trajectory, no controller state, no camera-rig parameters. The payoff is that you never write a separate controller for each viewpoint.

The representation itself is plain. With $T_t$ the camera-to-world pose at frame $t$:

In words, ΔT_t describes how far the camera has moved and turned at frame t relative to where it started, regardless of where the world's origin sits.

$T_0$ is the pose at the initial observation, $T_t$ the pose at frame $t$, and $\Delta T_t$ says "relative to where I started, how far have I moved and how far have I turned". It drops any dependence on where the global origin happens to sit, while keeping the magnitude and direction of translation and rotation intact (§4.2.1).

Some people still want to play with a keyboard. The key state $\mathbf{u}_t$ at time $t$ is converted into a 6-DoF increment using configured per-step translation and rotation magnitudes, then integrated:

$$\boldsymbol{\xi}_t = M\mathbf{u}_t,\qquad T_t = T_{t-1}\operatorname{Exp}\!\left(\widehat{\boldsymbol{\xi}}_t\right)$$

In words, the key state u_t is turned by the fixed table M into a 6-DoF increment (how far forward, how many degrees of turn), and that increment is applied on top of the previous pose to give the new pose T_t.

$M$ is a fixed lookup that translates key presses into "move this far forward, turn this many degrees", and $\operatorname{Exp}$ turns that increment into a change of pose. A continuous metric pose sequence skips the mapping and integration entirely and joins the same relative representation (§4.2.1). Whatever the entry point, the generator receives the same thing.

Why trajectories rather than raw action labels

The paper gives three reasons (§4.2.1). First, coverage: frame-aligned action logs exist only for instrumented gameplay and simulation, whereas camera trajectories can be recovered from a vastly broader pool of video once pose-quality filtering is applied. Second, the inverse mapping is ambiguous: recovering pseudo-actions from observed camera motion is generally not unique, and this is worst in third person, where character motion and camera-following behaviour jointly determine what you observe. Third, continuity: keyboards are discrete and system-specific, while trajectories preserve vertical motion, roll, motion magnitude, and translation–rotation coupling as they are.

The flip side is stated explicitly: the interface covers navigation and viewpoint control only. Semantic actor behaviours such as attacking, jumping, or object manipulation are not modelled by it (§4.2.1).

Getting the units right: one scale for the whole dataset

Relative poses fix the origin problem but not the translation-scale problem. The same number should mean a comparable physical camera motion across examples, and across heterogeneous sources it does not (§4.2.2).

The obvious fix is per-clip normalization, and the paper rejects it for two reasons: it erases the meaningful displacement differences between clips, and it introduces artificial scale changes — speed jumps — at continuation boundaries. Using the largest displacement in the corpus instead makes you hostage to reconstruction outliers.

So they estimate one robust scale from all metric training trajectories.

In words, each clip gets a number for its largest translation, and the 90th percentile of those numbers over the whole training set becomes the single scale s_global, a representative "about this much is the ceiling" value.

$m_i$ is the largest translation magnitude within clip $i$, and $Q_{0.9}$ is the 90th percentile over the training set — a single representative value for "about this much is the ceiling". Trajectories whose maximum displacement exceeds $s_{\mathrm{global}}$ are discarded rather than clipped, because clipping would alter the displacement magnitude itself. Everything retained is simply divided by $s_{\mathrm{global}}$, and rotations are left untouched (§4.2.2).

Figure 8 shows why this matters. The action-clean subset has per-clip maximum translations spread widely (mode 4.5, median 12.7), while the speech-present, audio-rich subset concentrates on much smaller motions (mode 0.5, median 3.9) (Figure 8). Normalizing each source independently would have flattened exactly that difference.

Putting relative geometry inside attention

How do you feed the trajectory to the model? The common approach is Plücker coordinates, representing each ray as a (direction, moment) pair. But the moment term contains the camera centre $\mathbf{o}_t$, so the values change with your choice of world origin. On top of that you have to materialize dense ray features and learn cross-frame relations with a dedicated camera encoder (§4.2.3).

EchoWM uses UCPE instead, which introduces relative ray geometry directly inside attention. The key is a simple identity: for any global rigid transform $G$, $(GT_i)^{-1}(GT_j)=T_i^{-1}T_j$ — the relation between two cameras is unchanged (§4.2.3). Make the query–key interaction depend only on the relative transform $\mathbf{G}_i\mathbf{G}_j^{-1}$ and the choice of coordinate frame stops mattering. Scale is handled separately by the calibration above, so the two concerns stay cleanly factored.

Implementation-wise, a lightweight camera-attention branch runs in parallel with the original video self-attention. Half of the channels in each head use the tiled ray transform; the rest keep spatiotemporal RoPE. The branch output is merged into the main path through a zero-initialized projection (§4.2.3). Because it starts at zero, the pretrained video pathway is preserved exactly at the start of trajectory fine-tuning, and the branch only ever learns the residual needed for camera conditioning. If the underlying question of how position enters attention is unfamiliar, Positional Encoding from Scratch is the groundwork.

One more design decision matters a lot: the camera branch operates on video tokens only. Audio tokens get no direct trajectory conditioning (§4.2.3). Sound is not produced by a dedicated action-to-sound controller; it is influenced only indirectly, through the existing cross-modal attention between video and audio. The paper is careful about this, saying it evaluates acoustic coherence under interaction "without claiming a separately supervised action-to-sound capability" (§6).

The data engine: keeping "what was commanded" apart from "what actually happened"

Training data comes from four complementary sources: internally collected gameplay, human-played Internet gameplay, Unreal Engine (UE) simulation, and general Internet video (§3.1). For the internal recordings, a ReShade-based plug-in suppresses HUD and interface overlays before capture, preserving the original field of view instead of cropping after the fact.

The most interesting choice is that action logs and camera trajectories are stored as two distinct signals. Issue a "forward" command and the character may not move at all if geometry blocks it. The action stream records control intent; the camera trajectory records the motion the environment actually realized (§3.1). The current model conditions only on the latter, but both are kept in the dataset.

Pose recovery uses ViPE as the optimization framework, combining VGGT-Omega for temporally coherent long-sequence geometry with MoGe-2 for per-frame metric-depth constraints (§3.3.1). ViPE is additionally adapted to optimize per-frame intrinsics, to cope with zoom, stabilization and cropping in Internet video. Crucially, recovery runs on roughly one-minute continuous windows rather than on the final short clips, so neighbouring clips inherit a shared reconstruction instead of being estimated in separate coordinate systems (§3.2). Retained sequences must then pass three separate checks: reconstruction and intrinsic-calibration reliability, temporal consistency (jitter, abrupt orientation changes), and motion plausibility (§3.3.3).

There is also a neat guard against textual motion leakage. Annotations are normalized into persistent fields (scene / style / viewpoint / subject) and time-varying ones (narrative / speech / sound). For trajectory-conditioned training the narrative field is removed — leave it in and the model will read the intended motion off the text instead of off the trajectory (§3.4). The treatment of silent sources is equally careful: UE renders have no audio stream, so they are treated as missing audio rather than semantic silence, and are never used as positive supervision for audio content (§3.4).

The four-stage curriculum

Training is staged (§4.3). Stage 1, AV-CPT, adapts the whole backbone $\theta$ on the audio-visual-rich mixture, with no trajectory conditioning yet. Stage 2, Action-SFT, freezes the backbone and trains only the camera-branch QKV projections and the zero-initialized output projection. It runs on the control-clean mixture, keeps only the static text fields, and disables the audio loss — constructing a situation in which the trajectory is the only thing that specifies subsequent camera evolution. Stage 3, Joint-FT, updates both parameter groups at a reduced learning rate on a smaller subset where trajectories and audio-visual content are both reliable.

The paper's own summary is the clearest statement of the principle: acquire each capability where its supervision is cleanest, then let the two pathways co-adapt (§6).

Post-training for long rollouts

A diffusion model normally attends bidirectionally over the whole sequence and denoises over many steps. To make it interactive you need two transformations: convert it to causal, chunk-wise computation, and compress the denoising into a few-step sampler that stays stable on self-generated history (§4.4). Diffusion Models from the Ground Up covers the underlying machinery.

The order is: audio-visual teacher forcing to make it causal, then short-horizon Self-Gradient Forcing (SGF) with DMD to make it few-step, then long-horizon SGF (§4.4). The trick in SGF is two forward passes: the first rolls out autoregressively under no_grad, using the same sampler and conditioning as real inference and recording the states; the second replays that record differentiably in parallel (§4.4.2). This restores supervision on the history the model wrote itself without backpropagating through the serial sampling process. The chunk-level causal mask is applied to five attention operations: video self-attention, audio self-attention, audio-to-video, video-to-audio, and UCPE attention.

The thing that bites hardest at long horizons is cache growth. A naive causal KV cache grows without bound as the rollout continues. EchoWM keeps a sink-plus-FIFO cache per modality:

In words, the history each modality attends to is the first S_m chunks (the sink) joined in order with the most recent R_m chunks, so it never exceeds S_m + R_m chunks however long the rollout runs.

The first group is the persistent prefix of $S_m$ chunks (the attention sink), the second is the most recent $R_m$ chunks, and $\Vert$ is ordered concatenation. Together they cap the directly visible history at $S_m+R_m$ chunks regardless of how long the rollout runs (§4.4.3). For the cache mechanism itself, see The KV Cache from Scratch.

The paper adds a subtle point: stacking layers gives an effective reach longer than that window. A FIFO representation at an upper layer has already absorbed context visible to that chunk at lower layers, so gradients can travel across layers and reach beyond the window of any single layer (§4.4.3).

Results: what is actually better, and by how much

On WBench Navigation (158 cases), EchoWM leads with an Average of 81.7 and a Consistency of 89.8 (Table 3). Its four-step causal variant, EchoWM-Flash, follows at 81.0 while taking the highest Interaction score of 87.9, slightly above the undistilled model's 87.2 (§5.1). The same table lists Genie 3 at 73.9, LTX-2.3 at 74.2, and Kling 3.0 at 79.0.

On the short-horizon SANA-WM-Bench evaluation (front 241 frames), EchoWM takes the highest VBench Overall on both splits — 83.91 on Simple and 83.96 on Hard. But SANA-WM has the lower rotation error on both (0.489° vs 0.523° on Simple, 1.248° vs 1.697° on Hard), and lower translation error on Hard as well (Table 4). Under the 961-frame protocol, rotation error grows from 0.523° to 3.223°, reaching 12.05° on the Hard split; the paper says plainly that accumulated pose drift remains the main challenge (§5.2). At that horizon LingBot-World also edges it out on VBench Overall (81.82 vs 81.36 on Simple).

Among causal models (Table 6), EchoWM-Flash takes the highest VBench Overall on both splits (80.13 and 81.06) along with the strongest trajectory accuracy and revisit consistency of the causal methods evaluated. Some baselines show a smaller $\Delta$IQ — the imaging-quality gap between the first and last 10-second windows — but the paper cautions that a small drop can simply reflect consistently low imaging quality rather than better preservation (Figure 14).

The human study runs on a self-built 200-case benchmark (WMB). EchoWM takes the largest share of overall-preference votes against both competitors: 46.50% versus LingBot-World-v2 and 63.13% versus HappyOyster (Table 7). The weaknesses are reported just as plainly. Against HappyOyster, semantic following goes the other way — 13.63% for EchoWM versus 29.44%. Broken down by scene, EchoWM receives only 8.67% of semantic-following votes in embodied-robot scenes and 2.50% in natural-outdoor scenes, despite leading overall preference in those same groups (§5.3).

The generalization result is worth noting: the training mixture contains no 2D or 2.5D game data and no robot data, yet the paper reports qualitatively coherent viewpoint evolution and preserved scene structure in exactly those unseen domains (§5.4).

Limitations, as the paper states them

Section 7 is honest. First, the model covers only navigation and viewpoint actions expressible as a relative 6-DoF trajectory; jumping, attacking, manipulation and robot commands are not explicitly represented, and it implements none of the rules, collision state or deterministic transitions of a real game engine. Second, there is no explicit persistent 3D memory, so geometry, subject identity, world state and audio can drift across repeated continuation turns. Third, the interaction range is restricted to the trajectories retained by the global filter — generalization to substantially larger translations, higher speeds, or unusual rotations is not established. Fourth, poses recovered from Internet and gameplay video remain imperfect even after filtering, and those errors may correlate with scene content or motion blur.

Read practically, the third point means a demo that looks great tells you little about your own speed regime. Test with trajectories from your actual operating range before drawing conclusions.

Using this in practice

Who touches this, and when. Technical artists in game and film pre-production, probing what a location would look and sound like before anyone builds it. Simulation engineers in robotics or driving who need footage — with audio — for situations that are hard to capture. And research engineers running a video-generation stack who have to decide how camera control gets added. For that last group this paper is mainly a design-decision artifact: Plücker plus a dedicated encoder, or relative geometry inside attention.

Parameters you will actually touch. The dataset-wide translation scale $s_{\mathrm{global}}$ (change it and the physical meaning of every input number changes with it); the per-step translation and rotation magnitudes inside the key-to-6-DoF map $M$; the sink count $S_m$ and recent-history count $R_m$; the number of sampler steps; and the tail window width carried across multi-turn continuation. On the preprocessing side: ViPE / VGGT-Omega / MoGe-2 for pose recovery, and the structured annotation field names (scene / style / viewpoint / subject / narrative / speech / sound).

Pitfalls that will bite you, six of them drawn from the paper:

1. Normalize trajectories per clip and speed will jump at continuation boundaries. Keep one scale for the whole corpus (§4.2.2).
2. Leave motion-describing text (the narrative field) in the condition and the model reads motion off the prose instead of the trajectory. This is the worst failure mode available: benchmarks look fine while control does nothing (§3.4).
3. Supervise on raw action logs and you teach the model a lie every time a forward command was blocked by a wall. Use the realized trajectory (§3.1).
4. Do not train silence from audio-less sources as if it were meaningful quiet. Treat it as missing (§3.4).
5. WBench adapts trajectory scale for robust open-domain scoring, which can hide errors in absolute motion magnitude. Measure speed fidelity separately (§5.5.1).
6. A small $\Delta$IQ does not mean the model holds up over long rollouts — it can just mean the imaging quality was low throughout (§5.2).

How to answer when asked. "Why UCPE rather than Plücker?" Because the Plücker moment term contains the camera centre and therefore depends on the world origin, and because it needs dense ray features plus a separate encoder; making the interaction depend only on the relative transform $\mathbf{G}_i\mathbf{G}_j^{-1}$ is origin-invariant and stays inside attention. "How do you stop the KV cache growing during long generation?" Sink-plus-FIFO fixes the directly attended history at $S+R$ per layer, and stacking causal layers buys effective reach beyond that window.

Wrapping up

EchoWM's claim in one line: stop building a control interface per subject, and fold everything into a single statement of observational intent. To make that work it represents pose relatively, pins translation to one dataset-wide scale, moves relative geometry inside attention, and trains each capability where its supervision is cleanest. Audio stays part of the jointly generated world rather than being bolted on by a controller.

On the numbers, it is strong on short-horizon visual quality and on interactive benchmarks, and it carries clear weaknesses in long-horizon pose drift and in semantic following on some scene types. The paper does not hide either. Read it as the current waypoint on the trajectory it sketches in §8 — from generated content that is only watched, toward worlds that can be continuously entered and influenced.