Paper Walkthrough: PhiZero — A World Model That Reasons in a Language of Physics Before It Renders

A close read of PhiZero, which reasons about how the world will change in a discrete 'physical language' learned self-supervised from video, and only then renders the footage — instead of predicting pixels directly. The two-stage tokenizer and reasoner, the top spot on Physics-IQ Verified, zero-shot motion transfer, and the limitations the paper itself admits.

PhiZero: A World Model Built Around Physical Language

Video AI can see, but it isn't observing

Ask a video generator to continue a clip of a tennis ball hitting a rubber duck. The ball arcs beautifully, the lighting and shadows are flawless — and the duck it slams into doesn't move at all. This failure mode, gorgeous pictures with broken physics, is common in today's video generation models. Figure 4 of this paper shows a baseline model doing exactly that.

The paper's diagnosis: mainstream video world models are trained to predict future frames directly in pixel space, which leaves the dynamics — how the world changes — implicit, buried inside a high-dimensional visual representation (§1). Getting better at reproducing appearance doesn't fix it, because the laws of change are never handled explicitly, and out come endings that don't add up physically.

PhiZero (Institute of Automation, Chinese Academy of Sciences, arXiv:2607.28624) attacks the problem with a reason-then-render design. The key piece is what the paper calls physical language: a sequence of discrete symbols that encodes nothing but how the state of the world changes.

An analogy: let the commentator call the play first

When we predict the future, we don't play back full-HD video in our heads. We read ahead at the level of events — "ball connects → duck topples → water ripples" — and fill in the visual detail afterward. The paper points to exactly this human capacity, abstracting predictively useful structure out of visual experience and reasoning over it explicitly in something language-like, as its inspiration (§1).

Natural language, though — English, Japanese, any of it — is too coarse to express the complicated state transitions of the physical world, the paper argues. "The cup tips over" doesn't pin down which way, how fast, or how the water spreads. So PhiZero builds a purpose-made language at an intermediate resolution — finer than natural language, coarser than pixels — from scratch, by self-supervised learning on unlabeled web video. Like a commentator turning a match into words, it turns only the change in a video into a string of symbols. That is physical language.

The big picture: split reasoning from rendering

PhiZero has two components (§3). Write the future video as $\mathbf{V}$, the first frame representing the current state as $I^{0}$, the action intent given as text as $c$, and the physical-language symbol sequence as $\mathbf{z}$. Future prediction then factors like this:

Put in words: the bar $\mid$ reads "given," so the left-hand side is the chance of ending up with this video and this particular string of change-symbols, starting from the one photo $I^{0}$ and the one instruction $c$. The right-hand side splits that single chance into two multiplied together — the chance of coming up with the symbol string from the photo and the instruction, times the chance of drawing this video once the symbol string is fixed.

Two stages, in other words: from a single current image and an instruction, first reason out how the world will change as a symbol sequence $\mathbf{z}$, then render the video from that sequence plus the one first frame (§3.1, Eq. 1). Separating the reasoning about change from the synthesis of pixels turns dynamics into an explicit object of inference.

The machine that makes physical language: the tokenizer

The first hard problem is learning the "words" themselves. That's the job of the Physical Language Tokenizer (§3.2). A video is first converted by a spatiotemporal encoder into a sequence of latent states; then, for each pair of adjacent states, a Q-Former module plays interrogator and extracts "what changed over this one interval" through 32 learnable queries. Taking change pair by pair rather than across the whole clip is a local inductive bias — it keeps the amount of change any single symbol has to carry small.

The extracted change features are turned into discrete symbols by FSQ (finite scalar quantization). It does nothing more than round each dimension to a fixed number of levels (8, 5, 5, 5, 5, 5 in the paper), and the combinations automatically define a vocabulary of roughly 25,000 words (§4.1). A 33-frame, four-second video becomes 9 latent states, and 8 adjacent pairs × 32 symbols yields a physical-language sequence of just 256 symbols.

The quickest way to get a feel for this is to think about JPEG compression. JPEG also shrinks an image by keeping only what a human can see and rounding the numbers coarsely. In the figure below, see for yourself how much information quantization can throw away while the picture still holds together. What PhiZero does is this same trick applied not to appearance but to change.