Paper Explained: VGI-Bench — Can Video Models Get the Middle of the Story Right?

A ground-up walkthrough of VGI-Bench, a benchmark that treats video generation models as machines that reason in pictures. Even the strongest model, Seedance 2.0, scores only 51.0 — and self-correction during denoising almost never happens.

VGI-Bench: Probing Visual Intelligence in Video Generation Models

Grading the working, not the answer

On a math test, an answer sheet with the right number and nothing else might just be a lucky guess. That is why graders look at the working.

There is now a push to do the same thing to video generation models. Give one of these models an image and a sentence, and it renders what happens next. Hand it a picture of a maze and the instruction "get to the goal," and if the video traces a valid path, hasn't the model solved the maze? The paper frames this as a shift in how we see these systems: from passive simulators toward visual reasoners that express their reasoning through generated frames (§1).

The hard part is grading. A final frame that looks correct means nothing if an object teleported halfway through. You need a way to grade the working, not the answer. VGI-Bench is a benchmark built squarely around that problem.

Three holes in the existing benchmarks

The paper identifies three gaps in prior work (§1).

The inputs don't look like the training distribution. Many benchmarks use line art or schematic diagrams because they are easy to author and control. But video models are trained on photorealistic footage. Abstract inputs produce more collapse and more ignored constraints — which may say more about visual domain mismatch than about reasoning ability.

Nothing forces the model to roll anything out. A lot of visual-QA-style tasks can be answered by staring at the input image. They never test whether a model can solve something by simulating how the scene evolves.

Difficulty runs wild. Once you mix in long-horizon tasks that no few-second video could express, or knowledge-heavy tasks requiring non-visual expertise like medicine, a failure tells you nothing. A benchmark everyone solves and a benchmark everyone fails carry exactly the same amount of information: none. For diagnosis, the tasks have to sit on the current capability boundary.

That first point — distribution mismatch — is the oldest trap in machine learning. Play with the figure below and watch training error and test error pull apart. When a benchmark's inputs fall outside the training distribution, the number you are reading is the one on the far side of that gap.

How VGI-Bench is built

VGI-Bench consists of 27 tasks and 810 instances (§3). Every task shares one I/O format: a text prompt plus an input image as the first frame, out of which the model generates a video — the image-to-video setting.

Tasks are organized by a two-level taxonomy (§3.1). The first level assigns each task to one of four mutually exclusive domains: Visual Organization (arranging, grouping, or selecting by visual attributes), Physical Manipulation (moving, placing, stacking, tool use), Structured Puzzles (transforming a state under explicit rules), and Spatiotemporal Dynamics (how states evolve, including ordering and temporal dependency). The second level attaches one or more non-exclusive skill tags — Spatial, Temporal, Planning, Attribute Grounding, Physics, Topology, Affordance. Domains tell you what the picture looks like; skills tell you what capability the task actually demands.

Each task is instantiated at three difficulty levels with roughly ten instances each. Input images are collected from the web and existing datasets, or generated by image models and then revised by hand until they match the intended design, all standardized to 16:9. Prompts specify the objects, the permitted actions, and the prohibited shortcuts, plus task-agnostic controls on background and camera motion so that irrelevant visual variation doesn't leak into the evaluation (§3.2).

The difficulty calibration step is the clever one (§3.3). For each proposed task, two of its easiest instances are run through models like Sora2, Veo3.1, and Kling3.0, and the task is kept only if at least one model solves it and at least one model fails it. The paper is careful to call this a sanity filter rather than a certificate of solvability.

Scoring: how far you got × how cleanly you got there

The score is the product of two metrics (§3.4). Completeness (Comp.) captures global progress toward the goal: a VLM judge views frames sampled uniformly at 2fps and maps the video to one of three tiers — complete, partial, or failed. Rubric Score (Rub.) captures local process validity by counting violations against a per-task checklist. Two mechanisms support it: adaptive sampling, which scans at 4fps, flags suspicious intervals, and re-inspects only those at 8fps; and a 10-frame sliding focus window, so the judge only ever looks at a short segment at a time.

Each checklist item is scored with an inverse decay in the violation count $x$.

In words, each checklist item starts at 1 point and shrinks as its violation count grows: 1.0 for none, 0.5 for one, 0.33 for two. The first violation costs the most, and each later one costs less.

Here $s_i$ is the score of the $i$-th checklist item and $x_i$ is how many times that item was violated. No violations gives 1.0, one gives 0.5, two gives 0.33. It encodes the intuition that the first violation hurts most and later ones add less. Rub. is the average across items. The final score combines them like this:

In words, the final score is goal progress multiplied by process cleanliness, so a video must do well on both to score well. If either Comp. or Rub. is low, the product is low, and that catches both the video that barely moves and the one that teleports to the goal.

Why multiply rather than add? A nearly static video can hardly break any rule, so its Rub. is high — but it never approaches the goal, so its Comp. is low. Conversely a video that teleports to the target state scores high on Comp. and low on Rub. Multiplication means either failure alone flattens the result.

The judge's own reliability is measured too (§4.3). The main configuration, using Gemini-3-Flash, reaches AUC 0.803 and 73.2% pairwise accuracy against human annotations. Removing adaptive fps drops it to 0.772 and removing the focus window to 0.753; swapping the judge model gives 0.690 for GPT-5-mini and 0.478 for Claude-Haiku-4.5. In practice, 73.2% is a long way from a judge that agrees with humans, so treating a few points of score difference as a verdict is unwise.

Results: 51.0 at the top

Evaluation uses half the instances per task under a fixed random seed (§4.1). On overall score (§4.2), Seedance 2.0 leads at 51.0, followed by MiniMax-H3 at 44.4, Kling3.0 at 44.0, Sora2 at 36.7, Veo3.1 at 32.0, Wan2.2 at 21.6, and HunyuanVideo-1.5 at 19.1. Commercial models consistently beat open-source ones, but the paper's conclusion is that none of the tasks are anywhere near solved.

Structured Puzzles is the hardest domain, and Topology and Temporal are the weakest skill dimensions. The breakdown isn't uniform, though: on Structured Puzzles alone, MiniMax-H3's 50.6 beats Seedance 2.0's 44.6. First place overall does not mean first place everywhere.

The paper also reports a strict success rate, counting an instance as a success only when both Comp. and Rub. are perfect (Appendix B.2). Overall, Seedance 2.0 gets 14.0% and HunyuanVideo-1.5 gets 1.0%. The human reference under the same criterion is 97.1%.

How they fail

Three recurring failure modes (§5.1). Physical collapse — unnatural deformation and object penetration; under goal or constraint pressure, physical causality gets sacrificed for a goal-shaped visual outcome. Rule violation — the video looks perfectly plausible while performing a prohibited action or skipping a required intermediate step. Object/state inconsistency — objects vanish, morph, or revert to an earlier state; motion is smooth but temporal state tracking is weak.

Input conditions matter too (§5.2). Even an oracle prompt that spells out the intended solution as explicitly as possible produces only limited gains. And converting the same task structure into a line-art variant noticeably reorders the models, an effect most pronounced among open-source systems.

How far does synthetic fine-tuning transfer?

The paper compares the released VBVR models — fine-tuned on a 1M-sample abstract-style dataset — against their base models (§5.3). Grouping tasks by how much their structure overlaps the training distribution is telling. For Wan2.2-I2V, the overlapping group jumps from 15.3 to 55.8 (+40.5), while the non-overlapping group only moves from 29.2 to 35.4 (+6.2). For Wan2.1-I2V, the non-overlapping group actually drops, from 25.6 to 22.0. The skill breakdown agrees: Planning rises from 10.4 to 44.1, while Temporal regresses from 37.0 to 25.8.

Per task, though, it isn't uniform. On untie_knot the success rate barely moves, yet the rubric score climbs from 0.02 to 0.34. The model didn't learn to solve it — it learned to stop thrashing.

When is the reasoning decided?

The most suggestive analysis comes last (§5.4). Diffusion models build their output by removing noise a little at a time. Work on diffusion language models has argued that later steps in this loop show self-correction: revising a wrong intermediate state. Does that happen in video?

The authors decode intermediate denoising states from four open-source models and label how the visible solution state changes between consecutive checkpoints. Wrong → correct (self-correction) stays below 1% everywhere and stops occurring entirely in later steps, while wrong → a different wrong runs an order of magnitude higher (23.1% at the 4→10 transition, 24.8% at 10→20). Once a state becomes readable it tends to stay put, with stability reaching 90.6% over the second half. Later denoising steps lock in and polish an early hypothesis, even one that already breaks the task's rules. The hope that "more steps will eventually converge on the right answer" is not supported by this measurement.

As an analogy, drag the optimization trajectory in the figure below. The path descends whichever basin it started in; it never crosses into another one.

Limitations the paper states itself

The Limitations section is explicit about four points. Every task is designed around the typical generation length of current models, roughly 5–10 seconds, so long-horizon assembly and long-trajectory planning are out of scope. Only image-to-video at a fixed 16:9 aspect ratio is covered. All prompts and rubrics are written in English. And the task suite is deliberately representative rather than exhaustive.

Using this in practice

When you're picking a model to ship. Citing the overall score as your reason is weak; the useful signal is the per-domain and per-skill breakdown. If you need a model to render an assembly procedure, read the Physical Manipulation and Planning columns. If you need it to respect board rules, read Structured Puzzles and Topology. Reversals are real — Seedance 2.0 leads overall but loses Structured Puzzles to MiniMax-H3.

When you build your own evaluation. The scoring architecture ports directly. Keep goal progress as a coarse three-tier discrete value rather than a continuous score. Turn rule violations into a checklist, count them, and weight the first one heavily with a monotonic decay like $1/(x+1)$. Multiply the two so that both "the output that never moves" and "the output that teleports" score badly. Split frame sampling by role — around 2fps for global progress, 4→8fps for violation detection — and keep the judging window near 10 frames.

Three ways this goes wrong. First, don't grade on the final frame alone; you will reward exactly the model that skipped the procedure and painted the target state. Second, re-measure human agreement whenever you swap the judge model for a cheaper one — the same procedure fell from AUC 0.803 to 0.478 in the ablation, and anything near 0.5 is a coin flip. The judge is the quietest thing in the pipeline to break. Third, don't judge synthetic fine-tuning by a single aggregate score: gains concentrate where the training distribution overlaps your task structure and can flatten or reverse where it doesn't.

None of this is specific to video. The same skeleton works for grading agent trajectories or long-form generation.

Related reading: how diffusion models turn noise into imagery in Introduction to Diffusion Models, a world model built around physical law in PhiZero, and the pitfalls of automated judging in Evaluating LLMs.