Paper Explained: WorldCrafter — A Video World Model That Remembers What You Saw, via Camera-Queryable Memory

Video world models tend to repaint a room differently when you walk back into it. WorldCrafter tackles this with an implicit 3D-aware memory that is read out from the viewpoint you are about to look from. We explain it from scratch with equations, interactive figures, and code.

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

What this paper does

The paper is "WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory" (arXiv:2609.24984, Peking University and Tencent ARC Lab).

In short, the abstract says: video world models let you explore a dynamic environment by generating new frames as you move the camera, but they struggle to stay faithful to what they showed you earlier, especially over long horizons and from new viewpoints. WorldCrafter addresses this by learning an implicit 3D-aware memory that can be queried by camera pose. The key idea is to let the requested viewpoint decide how multi-view evidence gets compressed into the generator's limited token budget. A memory encoder and a pose-conditioned readout module, trained jointly with the video generator, fold past observations into a fixed set of target-view tokens before denoising — without explicit depth-based correspondences. Combined with recent temporal context and few-step distillation, the system supports streaming exploration from a single image or a text prompt, and the authors report substantial gains in long-horizon consistency and camera-control accuracy while keeping visual quality over minute-scale exploration.

An analogy: the sofa that changes color

You leave a room with a red sofa, walk down the hallway, loop around, and come back. In real life the sofa is still red. A video generator that only remembers the last few seconds, though, may happily paint it blue.

Think of a reporter with a notebook that has a fixed number of pages. They can't transcribe everything they've seen, so before writing up their notes they check where they're headed next, and only copy out what matters for that destination. That is essentially what WorldCrafter does.

Intuition: three schools of memory

The paper sorts prior work into three families (§2.2).

1. Context memory: feed past frames directly into attention. Feeding all of them is expensive; feeding a selected few narrows what the model can see (§1).
2. Spatial memory: estimate depth and warp past frames into the target view. Great for revisits, but it relies on accurate geometry and struggles with moving objects (§2.2).
3. Implicit memory: compress history into a learned representation.

WorldCrafter belongs to the third family, but builds on LagerNVS, an encoder pretrained for novel-view synthesis (rendering a scene from a new angle). The authors argue that features from geometry estimators are weak on appearance, whereas novel-view synthesis can only be solved by keeping both shape and appearance — which is exactly what a memory needs (§2.2). If attention itself feels shaky, start with Attention from scratch.

Mechanism, part 1: chunk-by-chunk generation

Videos are too large to handle raw, so they're compressed into a latent $\mathbf{z}$ with a VAE (see VAE from scratch), and the model is trained with flow matching (§3.1).

In plain words: at $t=0$ you have the clean video, at $t=1$ pure noise, and anything in between is a blend. The model learns the velocity $\mathbf{v}^{*}_{t}$ pointing toward the noise, and generation walks that path backward.

Long videos are produced as a sequence of fixed-length chunks. The standard recipe only looks at recent frames $\mathbf{z}^{\mathrm{r}}$:

In words, it says: "use the last few frames and the text $\mathbf{y}$ as clues, and dig the current chunk out of the noise." Once the sofa leaves that short window, it's forgotten.

The real question is what to pack into a fixed token budget. The low-rank figure below lets you feel what happens when a budget shrinks.