Paper Walkthrough: Puffin-World — A World Model That Remembers Which Way Is Up

What changes when you give an image generator a sense of gravity? A ground-up walkthrough of Puffin-World's Omni-Camera representation and physics propagation, with equations and interactive figures.

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

What the paper claims

The original title is "Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States" (arXiv:2609.04196, published 3 September 2026, cs.CV), from a group centred on S-Lab at Nanyang Technological University.

Here is the abstract in plain terms. The authors propose Puffin-World, a unified multimodal architecture that folds physical understanding, spatial simulation, and 3D world generation and reconstruction into one model without leaning on external offline modules. It jointly models three native world states — physics (gravity field and latitude), geometry (depth), and appearance (image) — together with a unified Omni-Camera representation covering diverse tasks and flexible camera motion. It further introduces a strategy for propagating physical state into future frames, and argues that grounding absolute camera properties in the real world makes generation physically consistent and visually stable. Appearance and geometry are coupled inside a single generative process, so each future view is synthesised while its geometry is reconstructed. To reach scale, the authors build Puffin-16M — 15 million vision–language–camera triplets and 1 million trajectories — and state that code, models, and datasets have been released.

A metaphor: the camera operator who never loses "up"

Picture a camera assistant who is technically excellent but has no sense of balance. Told to "pan thirty degrees right," they execute perfectly — yet they do not know whether they are currently level or already leaning fifteen degrees. As pans and tilts accumulate the horizon creeps sideways, and by the end the ground runs diagonally across the frame.

Most current multi-view generators behave like that assistant. They handle the difference between viewpoints but carry no absolute pose with respect to the world. What Puffin-World does is hand the assistant a spirit level, then carry its reading through the entire sequence.

The intuition: grounding means knowing the angle between gravity and your line of sight

"Absolute pose" sounds imposing, but it amounts to this: keep gravity as a single 3D vector $\mathbf{g}$, and be able to measure the angle between it and the camera's viewing ray. A horizontal ray is perpendicular to gravity; look down and the angle sharpens, look up and it opens. Once you know that angle per pixel, everything follows — whether a pixel sits above or below the horizon, and which screen direction counts as up. The tool for measuring it is the dot product: large when two vectors point the same way, zero when perpendicular. Internalise that and every equation below reads easily.

The paper calls this angle latitude, because it measures how far each pixel's ray is lifted above the horizontal plane.

What was missing

The paper frames the gap in three strands (§1, §2). Generative world models predict each frame as RGB content with no explicit camera orientation or scene geometry. Unified multimodal models put understanding and generation in one network, but only for 2D semantics. And third, the relative camera representations such as Plücker embeddings that dominate camera-conditioned generation.

Relative representations encode inter-view relationships cleanly and fall out of structure-from-motion, but they have no anchor to the world, so the same relative trajectory can correspond to different absolute orientations. The consequence is orientation drift under long horizons, large rotations, and single-view settings. The datasets share the blame: because most 3D capture is handheld, roll is typically confined to $[-5^\circ,5^\circ]$ and pitch to $[-10^\circ,10^\circ]$ (§4.1.2). There was barely any rotation to learn from.

Three native world states

Puffin-World describes the world at three levels (§3.1.1). Physics is the gravity field and latitude map — the absolute cues that anchor an observation to the real world. Geometry is scene depth. Appearance is the RGB image. "Native" means these are perceived, propagated, and generated inside the same model rather than farmed out to external estimators.

The Omni-Camera representation: absolute and relative in nine channels

The core construct is the Omni-Camera representation (§3.1.2). For each pixel $\mathbf{x}=(u,v)$, an absolute camera field and a relative ray field are concatenated along the channel axis.

The same thing in words: nine numbers per pixel. The first three give absolute orientation with respect to the world (the in-image up-vector $\mathbf{u}_{\mathbf{x}}$ plus the latitude angle $\varphi_{\mathbf{x}}$); the remaining six give the relation to other views (ray origin ×3, ray direction ×3). Think of it as a nine-channel map at the image's own resolution. The relative ray direction $\mathbf{d}_{\mathbf{x}}$ follows from the intrinsics and extrinsics $\mathbf{K},(\mathbf{R},\mathbf{t})$ (paper Eq. 2), while the absolute side follows Perspective Fields, defined through the unit gravity direction $\mathbf{g}$.

There is the dot product you were just dragging around. The line in words: take the incoming ray direction against gravity, push it through an arcsine, and you get how far above the horizontal plane that pixel's ray points, as an angle. The up-vector $\mathbf{u}_{\mathbf{x}}$ is defined separately as the image-plane displacement observed when a 3D point is nudged against gravity and reprojected.