SolarWM: Training on 5 Seconds, Walking for an Hour — and Opening the Whole Stack

A data engine that folds 10 datasets into 1.43M clips under one contract, plus a three-stage recipe that adapts four different video backbones without erasing them. The causal models roll out for minutes to hours after training on 5-second sequences alone.

SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models

From video you watch to a world you enter

The paper here is SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models (arXiv:2609.02886, published 2026-09-02).

Its claim, in short: building an interactive video world model needs one continuous foundation running from data preparation all the way to long-horizon inference, and that is hard. Datasets disagree on temporal scale, camera geometry, visual quality, motion, and captioning style; video generators disagree on representation and architecture. Mix data naively and the supervision contradicts itself; write model-specific code and nothing is reproducible or comparable. SolarWM breaks that coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework. The engine converts 1.43 million canonical clips from 10 datasets into a unified, frame-aligned contract covering visual observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance — while keeping source processing separate from mixture construction. Under shared camera-conditioning, training, and inference interfaces, four models of 5B to 33B parameters are instantiated on Wan2.2, LTX-2.5, and MiniMax-H3, each keeping its native representations and objectives. Training is three stages: bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation. The resulting causal models support real-time interaction across rollouts from minutes to hours after being trained on 5-second sequences only.

An analogy: studio standards, and not ruining the actors

Two very different jobs live inside this paper.

The first is writing studio standards. Film arriving from ten places differs in length, camera log format, and exposure. You normalize it so every reel sits on the same shelf in the same shape. Crucially, you do not throw away the reels you dislike while normalizing. You keep them on a "rejected" shelf with a machine-readable reason attached, so that changing your mind later means pulling them back out rather than reshooting.

The second is directing without flattening the actors. Wan2.2, LTX-2.5, and MiniMax-H3 each arrive with acting ability earned on enormous amounts of video. You want to add one new instruction — "follow this camera" — without imposing a single vocal technique that erases what each of them was good at. SolarWM standardizes only how the instruction is delivered (the data and camera contract) and leaves the performance itself (latent representation, attention layout, optimization objective) to each backbone.

Where the difficulty actually is: two kinds of mismatch

The paper's framing is that data-side and model-side problems are coupled (§1).

On the data side, existing datasets differ in temporal scale, visual quality, motion distribution, captioning style, and camera conventions. Combine them naively and the supervision becomes inconsistent, so a model that looked fine on one source degrades under multi-source training. On the model side, video generators differ substantially in latent representation, attention structure, and conditioning mechanism. A shared adaptation strategy is desirable for scale, but ignoring that heterogeneity compromises pretrained capability — while backbone-specific adaptation kills systematic comparison and extensibility.

Hence the paper's starting point: the field still lacks a reproducible common foundation that handles multi-source data construction and backbone adaptation at the same time.

Read the paper in three layers

SolarWM has three layers, and separating them makes it go down faster.

Layer one is the data engine (§5): the machinery that turns 10 datasets into a single format where video, camera geometry, captions, quality metrics, accept/reject reasons, and provenance are all frame-aligned. This is where the bulk of the paper sits, and it reads less like a paper than like an operations manual.

Layer two is the model family (§6): Wan2.2 at 5B and 14B, LTX-2.5 at 22B, MiniMax-H3 at 33B, lined up under one camera-conditioning interface while each keeps its own latents, attention layout, and objective. The paper's own phrasing is that this is one framework with four routes, not four isolated releases.

Layer three is the training recipe (§4): build it bidirectionally, switch it to causal, finish with distillation.

The independence of these layers is the point. Change the data mixture and you touch no model code; add a backbone and you rerun no data processing. That is why the paper insists on the word engine rather than shipping a fixed list of training clips — what it wants to hand you is a device for rebuilding the mixture, not the mixture.

Why "trained on 5 seconds"

This part is background rather than a claim of the paper, but it makes everything downstream legible. Self-attention compares tokens against each other exhaustively, so compute and memory grow quadratically in sequence length. Training on an hour of video with full bidirectional attention was never on the table. That forces the shape of the solution: learn in a short window, then extend causally at inference. SolarWM training on 5 seconds and rolling out for an hour is an answer to that constraint.

The three stages, in outline

Before the equations, here is what each stage does (§4).

Stage one is bidirectional adaptation: take the pretrained video generator and acclimate it to camera-conditioned world data. The full training window is visible in both directions here — ordinary video-model training, except that camera control is injected at this point.

Stage two is teacher-forced autoregressive initialization: switch attention to causal, hide the future, show only the ground-truth past, and turn the model into a few-step autoregressive generator. The paper calls this stage fast because it is not relearning appearance and motion — it is activating causal prediction in a model that is already good.

Stage three is distribution matching distillation (DMD). Teacher forcing leaves a gap: training sees a clean past, while inference eats the model's own output as its past. DMD closes it by having the model actually roll out and then matching distributions.

The ordering carries information too. The paper states that most of the optimization belongs in stage one, that stage two converges quickly, and that DMD needs fewer steps still. From here we follow each stage through its equations, then open up the data engine underneath.