Paper Walkthrough: EchoWM — A World You Can Walk Into, and Hear, Driven by Camera Intent

Hand it a reference view and a stream of 'where I want to go', and it keeps generating 720p video together with environmental sound, music and speech. A walkthrough of EchoWM from the paper itself: one camera-intent interface for both first and third person, a single translation scale shared across the whole dataset, and the sink-plus-FIFO cache that makes long rollouts possible.

EchoWM: Open and Enterable Omnimodal World Models

The one-sentence version

EchoWM (arXiv:2608.23189) is a world model that takes a reference observation, a text description, and a continuous stream of navigation input, and keeps generating 720p video together with environmental sound, music and speech (Abstract). The defining move is that first-person and third-person scenes are driven by the same user-facing input: camera intent. It is a 42-page technical report from a joint team at HKUST, Peking University, JD's Joy Future Academy and others, with a project page and a code link listed on the paper.

A cinema, and a theme park

Start with the metaphor. Today's video generation is a cinema. You hand over a prompt like an order slip, take your seat, and watch whatever comes out. World models invert that relationship: they generate what happens next in response to continuous input from a user or an agent (§1). That is a theme park.

But, the paper continues, most existing world models either emit silent rollouts or expose a rigid action space tied to one particular game or controller (§1). And in an interactive world, audio is not decoration — it is information. Footsteps and collisions give physical feedback, ambient sound conveys spatial dynamics, and speech ties visible subjects to social and narrative context (§1).

Table 1 lays out the reported native capabilities of representative video and world models: high-resolution video, first- and third-person support, continuous trajectories, discrete keyboard control, environmental sound, background music, and spoken speech. EchoWM is the only entry with all seven marked (Table 1). Genie 3, for instance, is listed with discrete keyboard control but no audio; LTX-2.3 has all three audio capabilities but no participation interface at all. The capabilities exist — they are just scattered across different systems.

The authors call their target an enterable omnimodal world, and they fence it in from the start: interaction here means navigation and viewpoint evolution that can be written as a continuous trajectory, not unrestricted control over arbitrary subject behaviour (§1). That boundary stays consistent all the way through to the limitations section.

The difficulty is three kinds of mismatch

The paper names three coupled obstacles and assigns a design to each (§1).

The first is architectural: heterogeneous video sources differ fundamentally in metric scale and in control semantics, and naive normalization either erases displacement magnitude or degrades control consistency. The second is data: real-world video has rich acoustics but messy motion, while simulation has precise geometry, simple semantics, and no sound at all. The third is training: audio-visual quality and control responsiveness have their cleanest supervision in different subsets, and forcing them together induces distribution conflicts.

The three answers are, in order: a unified camera-intent interface, a four-source data engine, and a four-stage curriculum.

Camera intent as the single interface

Here is the core idea. Express user control not as a per-subject action vocabulary, but as how the observer intends to move and to look — camera intent (§1, §4.2.1).

In first-person scenes the trajectory directly drives observer ego-motion. In third-person scenes the same trajectory specifies camera evolution, while the associated character locomotion and camera-following behaviour are learned from data (§4.2.1). The model is given no explicit character trajectory, no controller state, no camera-rig parameters. The payoff is that you never write a separate controller for each viewpoint.

The representation itself is plain. With $T_t$ the camera-to-world pose at frame $t$:

$T_0$ is the pose at the initial observation, $T_t$ the pose at frame $t$, and $\Delta T_t$ says "relative to where I started, how far have I moved and how far have I turned". It drops any dependence on where the global origin happens to sit, while keeping the magnitude and direction of translation and rotation intact (§4.2.1).

Some people still want to play with a keyboard. The key state $\mathbf{u}_t$ at time $t$ is converted into a 6-DoF increment using configured per-step translation and rotation magnitudes, then integrated:

$$\boldsymbol{\xi}_t = M\mathbf{u}_t,\qquad T_t = T_{t-1}\operatorname{Exp}\!\left(\widehat{\boldsymbol{\xi}}_t\right)$$

$M$ is a fixed lookup that translates key presses into "move this far forward, turn this many degrees", and $\operatorname{Exp}$ turns that increment into a change of pose. A continuous metric pose sequence skips the mapping and integration entirely and joins the same relative representation (§4.2.1). Whatever the entry point, the generator receives the same thing.

Why trajectories rather than raw action labels

The paper gives three reasons (§4.2.1). First, coverage: frame-aligned action logs exist only for instrumented gameplay and simulation, whereas camera trajectories can be recovered from a vastly broader pool of video once pose-quality filtering is applied. Second, the inverse mapping is ambiguous: recovering pseudo-actions from observed camera motion is generally not unique, and this is worst in third person, where character motion and camera-following behaviour jointly determine what you observe. Third, continuity: keyboards are discrete and system-specific, while trajectories preserve vertical motion, roll, motion magnitude, and translation–rotation coupling as they are.

The flip side is stated explicitly: the interface covers navigation and viewpoint control only. Semantic actor behaviours such as attacking, jumping, or object manipulation are not modelled by it (§4.2.1).

Getting the units right: one scale for the whole dataset

Relative poses fix the origin problem but not the translation-scale problem. The same number should mean a comparable physical camera motion across examples, and across heterogeneous sources it does not (§4.2.2).

The obvious fix is per-clip normalization, and the paper rejects it for two reasons: it erases the meaningful displacement differences between clips, and it introduces artificial scale changes — speed jumps — at continuation boundaries. Using the largest displacement in the corpus instead makes you hostage to reconstruction outliers.

So they estimate one robust scale from all metric training trajectories.

$m_i$ is the largest translation magnitude within clip $i$, and $Q_{0.9}$ is the 90th percentile over the training set — a single representative value for "about this much is the ceiling". Trajectories whose maximum displacement exceeds $s_{\mathrm{global}}$ are discarded rather than clipped, because clipping would alter the displacement magnitude itself. Everything retained is simply divided by $s_{\mathrm{global}}$, and rotations are left untouched (§4.2.2).

Figure 8 shows why this matters. The action-clean subset has per-clip maximum translations spread widely (mode 4.5, median 12.7), while the speech-present, audio-rich subset concentrates on much smaller motions (mode 0.5, median 3.9) (Figure 8). Normalizing each source independently would have flattened exactly that difference.