Paper Explained: VBVR-Pro — Making "Thinking in Pictures" Trainable, Scorable, and Comparable

Native visual reasoning treats image and video generation as the medium of thought. VBVR-Pro supplies 300 procedurally generated tasks and deterministic scorers that replace the VLM judge, then pits image, video, and interleaved generation against each other on identical ground. Written for readers starting from zero.

VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

Some problems are solved by moving pictures in your head

Hand someone a block and ask what it looks like rotated 180 degrees, and they do not answer in words. They turn the block mentally, check the intermediate views, and arrive at a shape. Solving a maze works the same way: you trace routes with your eye, hit a dead end, back up. The work happens before anything becomes language.

Today's AI usually translates such problems into text anyway. It looks at an image, narrates it — "a red box at the upper left, a blue box at the lower right" — and reasons over that narration. Whatever falls out during translation is simply invisible to the language model downstream.

Native visual reasoning takes the opposite stance: the picture is neither input nor output but the paper you think on. In the paper's phrasing, visual states (images and videos) are "not merely inputs to be understood or outputs to be rendered, but first-class substrates for problem solving beyond language" (Abstract). Ask a video generator to produce a video of the maze being solved, and the model advances frame by frame, updating state as it goes. That generation process is the reasoning.

Three things were blocking progress: training data, scoring, and fair comparison

It is an appealing idea that had stalled. The paper sorts the blockage into three parts (§1).

One: there are no trainable tasks. Existing benchmarks are built for evaluation and ship little or no training data. Large-scale synthetic sources, meanwhile, lean toward simplified symbolic settings, leaving open whether anything learned there transfers.

Two: nothing scores it reliably. Who decides whether a generated video reasoned correctly? The prevailing answer is VLM-as-a-judge — hand the outputs to a strong multimodal LLM. But correctness in visual reasoning turns on exact counts, fine-grained spatial relations, temporal consistency, and rule satisfaction. That is precisely where judges are weak.

Three: the substrates were never compared fairly. Image generation, video generation, or interleaved text-image generation — which one actually supports visual reasoning? Almost no prior work trained and evaluated them under one task distribution and one verification protocol.

VBVR-Pro (arXiv:2608.26105, released 2026-08-26, a collaboration of more than 50 researchers and engineers) is proposed as a closed-loop testbed that closes all three at once.

Tasks written as programs

VBVR-Pro converts visual reasoning into a controlled space of 300 procedurally generated tasks (§2.2). Of these, 150 are reimplemented and revised from the earlier VBVR suite and 150 are newly designed.

A "task" here is not a problem but a program that manufactures problems. Each is a parameterized generator that samples a configuration from a structured parameter space — grid size, object count, layout, appearance, difficulty — and instantiates a matching instance. A task-specific solver computes the correct answer at the same time, so labelled data appears without any manual annotation.

Tasks are organized by five cognitive faculties (§2.1): perception (OCR, symbol recognition, counting, sorting, visual comparison), spatiality (relative position, 3D structure, navigation), transformation (operations such as 2D translation and rotation), abstraction (pattern induction, symmetry and shape completion, reasoning under newly specified rules), and knowledge (conventional icon meanings, physical phenomena, common sense, the rules and strategies of classic games, the mechanics of everyday objects). A task may carry several labels at once.

The other load-bearing design choice is multi-modality (§2.2). Every instance is rendered into aligned modalities: video, images, and an interleaved format with text annotations. Sampling frames out of the video is not enough, so the image side uses three output regimes. Last-Frame shows only the final state, for tasks whose answer fits in one picture. Key-Frame presents the states that capture the essential transitions — for path tasks such as maze navigation, the full solution trajectory is drawn onto the selected frames. Multi-Frame uniformly samples intermediate states for tasks that grade process integrity or temporal continuity. Videos and images are not separate corpora but different renderings of the same problem, which is exactly what makes the modality comparison fair.

On scale: roughly 3.47M images and 1.30M videos across the 300 tasks. Sampling 5,000 instances from each of 250 tasks yields 1.25M training instances, with a further 50K set reserved for RL. The remaining 50 tasks are never trained on and serve as out-of-domain (OOD) held-out tasks; VBVR-Pro-Bench pairs 50 in-domain (ID) tasks with those 50 OOD tasks for 100 tasks total (§2.3).

The 150 new tasks are visibly heavier than the reworked ones. Averaged over 50 sampled question frames per task, the median new task contains 80 distinct connected color regions against 12 for the reworked tasks. In a blinded pairwise comparison, the new task was judged to require deeper reasoning in 113 of 150 pairs, and the share requiring multi-step reasoning rose from 7% to 47% (§2.3).

Is it just memorization?

Train on procedurally generated synthetic data and of course you get better at that kind of problem. The real question is whether the ability travels. The paper runs several diagnostics against this suspicion (§1, §4.4).

One is a nearest-neighbor analysis. For benchmark cases the model got right, it embeds the initial image with CLIP and DINOv2 features and the instruction with BGE-base-en-v1.5, then retrieves the closest training instances. If success were rote recall, a near-twin should surface.

The paper reports the opposite. The retrieved training instances are visually and semantically distinct from the queries; some share a related instruction, but their appearance differs substantially (§4.4). The reading is that the model picked up reusable abstract visual operations — pattern finding, ordering, object manipulation — rather than surface templates.

Three ways the VLM judge breaks

Now the second pillar: verifiable reward scorers. The paper first takes the VLM judge apart with measurements (§3.1).

Break one: it misses fine-grained perception. In an additive color-mixing task the generated ball has the wrong hue, yet all three VLM judges assign at least 0.80. The proposed scorer gives 0.40.

Break two: it overlooks decisive errors. In a concentric-ring task the output alters circle sizes and colors that should have been preserved, and the judges still hand out scores like 0.78 and 0.93.

Break three: it misunderstands the task. In a pattern-completion task a correct answer earns 0.94 from the proposed scorer but only 0.24–0.80 from the VLM judges.

So the errors run in both directions — overrating wrong outputs and underrating right ones. Cost compounds the problem: RL scores an enormous number of rollouts across many steps, so proprietary judges burn API spend while open-source judges burn GPU hours that compete with training under a fixed budget.

Worst of all, the judges are not reproducible. Even at temperature 0, re-scoring the same videos changes the score for 54.6% to 92.8% of cases, with a maximum score change of 0.221 (Table 2). The proposed scorer changes 0.0% of cases, maximum change 0.000. Used as a reward, that nondeterminism is simply noise.