Paper Explained: VGI-Bench — Can Video Models Get the Middle of the Story Right?

A ground-up walkthrough of VGI-Bench, a benchmark that treats video generation models as machines that reason in pictures. Even the strongest model, Seedance 2.0, scores only 51.0 — and self-correction during denoising almost never happens.

VGI-Bench: Probing Visual Intelligence in Video Generation Models

Grading the working, not the answer

On a math test, an answer sheet with the right number and nothing else might just be a lucky guess. That is why graders look at the working.

There is now a push to do the same thing to video generation models. Give one of these models an image and a sentence, and it renders what happens next. Hand it a picture of a maze and the instruction "get to the goal," and if the video traces a valid path, hasn't the model solved the maze? The paper frames this as a shift in how we see these systems: from passive simulators toward visual reasoners that express their reasoning through generated frames (§1).

The hard part is grading. A final frame that looks correct means nothing if an object teleported halfway through. You need a way to grade the working, not the answer. VGI-Bench is a benchmark built squarely around that problem.

Three holes in the existing benchmarks

The paper identifies three gaps in prior work (§1).

The inputs don't look like the training distribution. Many benchmarks use line art or schematic diagrams because they are easy to author and control. But video models are trained on photorealistic footage. Abstract inputs produce more collapse and more ignored constraints — which may say more about visual domain mismatch than about reasoning ability.

Nothing forces the model to roll anything out. A lot of visual-QA-style tasks can be answered by staring at the input image. They never test whether a model can solve something by simulating how the scene evolves.

Difficulty runs wild. Once you mix in long-horizon tasks that no few-second video could express, or knowledge-heavy tasks requiring non-visual expertise like medicine, a failure tells you nothing. A benchmark everyone solves and a benchmark everyone fails carry exactly the same amount of information: none. For diagnosis, the tasks have to sit on the current capability boundary.

That first point — distribution mismatch — is the oldest trap in machine learning. Play with the figure below and watch training error and test error pull apart. When a benchmark's inputs fall outside the training distribution, the number you are reading is the one on the far side of that gap.

How VGI-Bench is built

VGI-Bench consists of 27 tasks and 810 instances (§3). Every task shares one I/O format: a text prompt plus an input image as the first frame, out of which the model generates a video — the image-to-video setting.

Tasks are organized by a two-level taxonomy (§3.1). The first level assigns each task to one of four mutually exclusive domains: Visual Organization (arranging, grouping, or selecting by visual attributes), Physical Manipulation (moving, placing, stacking, tool use), Structured Puzzles (transforming a state under explicit rules), and Spatiotemporal Dynamics (how states evolve, including ordering and temporal dependency). The second level attaches one or more non-exclusive skill tags — Spatial, Temporal, Planning, Attribute Grounding, Physics, Topology, Affordance. Domains tell you what the picture looks like; skills tell you what capability the task actually demands.

Each task is instantiated at three difficulty levels with roughly ten instances each. Input images are collected from the web and existing datasets, or generated by image models and then revised by hand until they match the intended design, all standardized to 16:9. Prompts specify the objects, the permitted actions, and the prohibited shortcuts, plus task-agnostic controls on background and camera motion so that irrelevant visual variation doesn't leak into the evaluation (§3.2).