PAWBench Explained — Can Video Generators Get the Odds Right, Not Just the Physics?

If a video generator is a world model, it owes you more than one plausible rollout — it owes you the right distribution over futures. PAWBench measures that probabilistic alignment across 50 scenarios and 11 systems, and finds that no model gets all the requirements at once.

PAWBench: How Far Are We from Probabilistically Aligned World Modeling?

One plausible future is not enough

Someone shows you a single video of a coin toss. It lands heads, the motion looks right. Does that mean the model understands physics?

If you run the same image and the same prompt 100 times and get heads 100 times, the answer is no. Heads is a perfectly valid outcome. What is wrong is the proportion.

That single point is what the PAWBench paper is about. Video generators are increasingly described as world models, yet existing evaluation still scores generated videos one at a time — visual quality, temporal coherence, instruction following, physical plausibility (§1). However natural each individual clip looks, none of that tells you whether the distribution the model draws from, under a fixed condition, is correct. The paper names this distribution-level requirement probabilistic alignment and makes it measurable.

The analogy: judge the frequencies, not the roll

Nobody evaluates a die by rolling it once and saying "it came up six, and it rolled nicely." You roll it many times and check whether 1 through 6 come up in roughly equal proportion.

That is exactly the move here. Hold the source image $x$ and the action prompt $a$ fixed, sample the same model repeatedly, convert each video into a label for "what finally happened," and count frequencies (§3.1). Instead of comparing videos frame by frame, you first reduce them to tallies of outcomes.

Two levels of "aligned"

The paper splits "the distribution is right" into two levels (§2).

• Support alignment: can the model produce every valid outcome, without dropping any?
• Probability-mass alignment: do those outcomes appear in the right proportions?

The ordering of strength is: one plausible rollout < support alignment < probability-mass alignment. In the paper's own phrasing, a model can be diverse without being aligned — covering every outcome does not guarantee assigning the correct probability to each.

In the figure below, keep the number of outcomes fixed and watch how differently the probability can be spread across them.

The mechanism: from trajectories to tallies

Write $x$ for the initial observation, $a$ for the action, and $\tau$ for a generated trajectory (a video). A video world model $M$ is then the conditional distribution $P_{M}(\tau\mid x,a)$ (§2). Trajectories are far too high-dimensional to compare directly, so the paper introduces a map $g$ from a trajectory to its terminal outcome. With $\mathcal{Y}$ the finite set of valid terminal outcomes,

Read that in words: it is "the fraction of generated videos whose outcome is $y$." Here $y$ is a visible result such as heads or tails, and $g$ is the judge that watches a video and attaches that label. Differences between trajectories that end the same way are deliberately thrown away.

When a reference distribution $q$ (the correct proportions) is available, the paper compares with total variation distance (TVD), which is half the $\ell_1$ distance between two categorical distributions (§3.3).

Again in words: add up the probability gaps over all outcomes and halve the total. An exact match scores 0; a model that produces 70/30 where 50/50 is correct scores 20 on the $\times 100$ scale used in the paper (Tab. 1 caption). When $q$ cannot be justified, the paper measures what fraction of the valid outcomes actually showed up instead — coverage.

How PAWBench is built

Fifty scenarios span eight physical mechanism groups and split into two tracks (§3.1). PAW-Calibration (25) covers scenarios whose reference distribution $q$ follows from analysis or symmetry — tossing, rotation, routing and draw-style randomizers. PAW-Coverage (25) covers scenarios whose valid outcomes can be enumerated but whose relative probabilities cannot be justified — collision, stability, agent interaction and material transition.

The split exists to separate two failures that single-sample evaluation cannot tell apart. The first track measures misallocated probability (every rollout looks plausible, yet one outcome occurs far too often); the second measures missing outcomes (each clip looks fine, but some valid futures never appear at all). That is why the paper refuses to fold them into one score.

Scenario construction carries three requirements of its own (§3.2). Stochasticity must come from a visible physical mechanism, not from an ambiguous prompt or hidden initial conditions. The action prompt must specify a single atomic intervention whose completion is judgeable from the video. Terminal outcomes must form a finite, visually distinguishable set. All 50 scenarios were curated and reviewed by hand, and reference distributions and failure criteria were fixed before any model was evaluated.