H3-World, Explained — Turning Language Understanding into World Control

How to turn a 33B video generator into a controllable world model without bolting on an action module: express actions as sentences, bind each sentence to a video-latent interval, and use attention routing to stop the crosstalk. Only 0.199% of parameters are trained.

H3-World: Turning Language Understanding into World Control

A movie machine is not a game console

A video generator is a machine for making short films. You hand it a sentence, and something reasonably coherent comes out. But once the film exists, you cannot lean in halfway through and say "turn a bit more to the left."

A game console is different. The world reacts the moment you press a key, by the amount you pressed, at the time you pressed it. A model that has learned that reaction is a world model. The question this paper takes on is how to convert an existing, very large film machine into a game console.

The paper is "H3-World: Turning Language Understanding into World Control" (arXiv:2609.01560, published 1 September 2026; Danze Chen, Zeqing Wang, Ziyue Lin, Xingyi Yang, Yeying Jin — Tencent / National University of Singapore / The Hong Kong Polytechnic University).

Here is the abstract's claim in plain terms. H3-World is an efficient framework that turns the 33B MiniMax-H3 video generator into an interactive world model. The authors' key finding is that as large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3 already supports zero-shot control of character behaviour and camera motion through natural-language instructions. H3-World turns that coarse language interface into precise, temporally grounded world control without introducing dedicated action modules. Each action is represented as a structured combination of character and camera instructions, aligned with the corresponding temporal video latents. Temporal attention routing then restricts each instruction to its intended time interval and reduces control leakage across actions. With only 8,000 gameplay samples, 10,000 LoRA optimisation steps and 0.199% trainable parameters, H3-World achieves effective character and camera control while preserving generation quality, and it generalises to unseen scenarios.

Generation is not control

The paper's opening framing is exact: generation is not control (§1). A pretrained video model does not, by itself, expose the precise action interface that interactive world modelling requires.

Existing systems therefore bolt on new control pathways — discrete action embeddings, conditional modules, explicit camera geometry, or direct adaptation of the backbone (§1). These are typically learned from temporally aligned action-video trajectories, so they demand extra supervision and adaptation on top of the pretrained generator. They add computation and storage, and extensive adaptation can disturb capabilities acquired during pretraining. More fundamentally, the paper argues, this paradigm treats control as something that must be learned largely on top of the pretrained model.

But a large video generator may already contain part of that bridge. MiniMax-H3, with no action-conditioned training at all, responds to coarse textual motion instructions with roughly correct behaviour and realistic visual dynamics (§1, Figure 1). Language already acts as a coarse control interface for strong text-to-video models — that observation is the starting point.

Text and video already sit in the same sequence

The design only makes sense once you know MiniMax-H3's architecture. It processes text, image, audio and video tokens in a shared sequence through single-stream self-attention. There is no separate text cross-attention layer, and the model jointly denoises the complete future horizon (§2.2, §3.1).

So conditions (sentences) and generation targets (video) are already taking dot products with each other on the same playing field. If that is true, designing control is not a matter of adding a layer. It is a matter of deciding, inside that one sequence, who is allowed to read whom.

Turning actions into sentences (§3.2)

External controls are first represented as discrete control states. Each state contains eight recorded character and camera keys plus one binary camera-speed flag. During training the speed flag is derived from the estimated camera yaw rate — how fast the view swings left or right; at inference the user specifies it directly (§3.2).

Then comes a granularity problem. One native H3 video latent covers a short interval of RGB frames. So the recorded key states within that interval are aggregated into a single latent-level state, with a key marked active if it occurs in any frame of the interval. Opposing keys — left and right pressed together — are cancelled before the prompt is built.

Symbols alone carry little information about the visual transition they should produce. To expose the compositional structure of the control space, the paper separates character control from camera control. For the $k$-th latent interval, the action is

Stated in words, the momentary control at step $k$ is a pair: what the character does ($u_k$) and what the camera does ($c_k$). $\mathcal{U}$ holds the character-control commands and $\mathcal{C}$ the camera-control commands — nine character clauses and sixteen camera clauses. Each pair maps to a short textual instruction.

That is the line which says: make the $k$-th control into a single sentence, a character clause followed by a camera clause. $\mathcal{T}$ is "rewrite a command as a fixed-phrasing clause" and $\|$ is clause concatenation. A backward-left character command combined with a slow right-pan camera command produces "the man walks backward and strafes left, camera pans right slowly." Across the dataset, character and camera clauses follow a shared grammatical structure. This places external controls in H3's native text-conditioning space while preserving the factorisation of the original action space.