Paper Explained: VBVR-Pro — Making "Thinking in Pictures" Trainable, Scorable, and Comparable

Native visual reasoning treats image and video generation as the medium of thought. VBVR-Pro supplies 300 procedurally generated tasks and deterministic scorers that replace the VLM judge, then pits image, video, and interleaved generation against each other on identical ground. Written for readers starting from zero.

VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

Some problems are solved by moving pictures in your head

Hand someone a block and ask what it looks like rotated 180 degrees, and they do not answer in words. They turn the block mentally, check the intermediate views, and arrive at a shape. Solving a maze works the same way: you trace routes with your eye, hit a dead end, back up. The work happens before anything becomes language.

Today's AI usually translates such problems into text anyway. It looks at an image, narrates it — "a red box at the upper left, a blue box at the lower right" — and reasons over that narration. Whatever falls out during translation is simply invisible to the language model downstream.

Native visual reasoning takes the opposite stance: the picture is neither input nor output but the paper you think on. In the paper's phrasing, visual states (images and videos) are "not merely inputs to be understood or outputs to be rendered, but first-class substrates for problem solving beyond language" (Abstract). Ask a video generator to produce a video of the maze being solved, and the model advances frame by frame, updating state as it goes. That generation process is the reasoning.

Three things were blocking progress: training data, scoring, and fair comparison

It is an appealing idea that had stalled. The paper sorts the blockage into three parts (§1).

One: there are no trainable tasks. Existing benchmarks are built for evaluation and ship little or no training data. Large-scale synthetic sources, meanwhile, lean toward simplified symbolic settings, leaving open whether anything learned there transfers.

Two: nothing scores it reliably. Who decides whether a generated video reasoned correctly? The prevailing answer is VLM-as-a-judge — hand the outputs to a strong multimodal LLM. But correctness in visual reasoning turns on exact counts, fine-grained spatial relations, temporal consistency, and rule satisfaction. That is precisely where judges are weak.

Three: the substrates were never compared fairly. Image generation, video generation, or interleaved text-image generation — which one actually supports visual reasoning? Almost no prior work trained and evaluated them under one task distribution and one verification protocol.

VBVR-Pro (arXiv:2608.26105, released 2026-08-26, a collaboration of more than 50 researchers and engineers) is proposed as a closed-loop testbed that closes all three at once.

Tasks written as programs

VBVR-Pro converts visual reasoning into a controlled space of 300 procedurally generated tasks (§2.2). Of these, 150 are reimplemented and revised from the earlier VBVR suite and 150 are newly designed.

A "task" here is not a problem but a program that manufactures problems. Each is a parameterized generator that samples a configuration from a structured parameter space — grid size, object count, layout, appearance, difficulty — and instantiates a matching instance. A task-specific solver computes the correct answer at the same time, so labelled data appears without any manual annotation.

Tasks are organized by five cognitive faculties (§2.1): perception (OCR, symbol recognition, counting, sorting, visual comparison), spatiality (relative position, 3D structure, navigation), transformation (operations such as 2D translation and rotation), abstraction (pattern induction, symmetry and shape completion, reasoning under newly specified rules), and knowledge (conventional icon meanings, physical phenomena, common sense, the rules and strategies of classic games, the mechanics of everyday objects). A task may carry several labels at once.

The other load-bearing design choice is multi-modality (§2.2). Every instance is rendered into aligned modalities: video, images, and an interleaved format with text annotations. Sampling frames out of the video is not enough, so the image side uses three output regimes. Last-Frame shows only the final state, for tasks whose answer fits in one picture. Key-Frame presents the states that capture the essential transitions — for path tasks such as maze navigation, the full solution trajectory is drawn onto the selected frames. Multi-Frame uniformly samples intermediate states for tasks that grade process integrity or temporal continuity. Videos and images are not separate corpora but different renderings of the same problem, which is exactly what makes the modality comparison fair.

On scale: roughly 3.47M images and 1.30M videos across the 300 tasks. Sampling 5,000 instances from each of 250 tasks yields 1.25M training instances, with a further 50K set reserved for RL. The remaining 50 tasks are never trained on and serve as out-of-domain (OOD) held-out tasks; VBVR-Pro-Bench pairs 50 in-domain (ID) tasks with those 50 OOD tasks for 100 tasks total (§2.3).

The 150 new tasks are visibly heavier than the reworked ones. Averaged over 50 sampled question frames per task, the median new task contains 80 distinct connected color regions against 12 for the reworked tasks. In a blinded pairwise comparison, the new task was judged to require deeper reasoning in 113 of 150 pairs, and the share requiring multi-step reasoning rose from 7% to 47% (§2.3).

Is it just memorization?

Train on procedurally generated synthetic data and of course you get better at that kind of problem. The real question is whether the ability travels. The paper runs several diagnostics against this suspicion (§1, §4.4).

One is a nearest-neighbor analysis. For benchmark cases the model got right, it embeds the initial image with CLIP and DINOv2 features and the instruction with BGE-base-en-v1.5, then retrieves the closest training instances. If success were rote recall, a near-twin should surface.

The paper reports the opposite. The retrieved training instances are visually and semantically distinct from the queries; some share a related instruction, but their appearance differs substantially (§4.4). The reading is that the model picked up reusable abstract visual operations — pattern finding, ordering, object manipulation — rather than surface templates.

Three ways the VLM judge breaks

Now the second pillar: verifiable reward scorers. The paper first takes the VLM judge apart with measurements (§3.1).

Break one: it misses fine-grained perception. In an additive color-mixing task the generated ball has the wrong hue, yet all three VLM judges assign at least 0.80. The proposed scorer gives 0.40.

Break two: it overlooks decisive errors. In a concentric-ring task the output alters circle sizes and colors that should have been preserved, and the judges still hand out scores like 0.78 and 0.93.

Break three: it misunderstands the task. In a pattern-completion task a correct answer earns 0.94 from the proposed scorer but only 0.24–0.80 from the VLM judges.

So the errors run in both directions — overrating wrong outputs and underrating right ones. Cost compounds the problem: RL scores an enormous number of rollouts across many steps, so proprietary judges burn API spend while open-source judges burn GPU hours that compete with training under a fixed budget.

Worst of all, the judges are not reproducible. Even at temperature 0, re-scoring the same videos changes the score for 54.6% to 92.8% of cases, with a maximum score change of 0.221 (Table 2). The proposed scorer changes 0.0% of cases, maximum change 0.000. Used as a reward, that nondeterminism is simply noise.

Putting scoring back on rules

A dedicated scorer is implemented for each of the 100 tasks in VBVR-Pro-Bench (§3.2). The core design decision is to stop comparing pixels and compare task-relevant semantic entities instead.

Classical computer vision does the extraction: HSV color segmentation, contour detection, OCR for rendered digits and labels, and trajectory tracking across frames pull out each object's color, shape, position, and count. Grading then happens over those attributes. A pixel-level comparison would penalize a correct answer for a slight spatial shift or a difference in rendering style; attribute-level verification only asks whether the task was solved.

The checks are aggregated two ways. Writing the paper's description as formulas: tasks with soft criteria use a weighted sum,

In words, the soft score is a weighted sum of the check values: each check contributes in proportion to its weight, so a near-miss still earns a fair amount of partial credit.

while tasks with hard constraints combine multiplicatively.

In words, the hard score is the product of all the check values, so if any single check is close to zero, the whole score collapses toward zero. This is what you use when a violation, such as a maze solution passing through a wall, must get no partial credit.

Here $c_i$ is how well check $i$ was satisfied, on a 0-to-1 scale, and $w_i$ is a hand-calibrated weight. Equation (1) adds up partial credit, so a near-miss still scores something. Equation (2) multiplies, so a single value near zero collapses the whole score. That is the tool you reach for when you refuse to give partial credit to a maze solution that walked through a wall.

In implementation terms:

attrs  = extract_entities(frame)          # HSV segmentation, contours, OCR, tracking
checks = [c(attrs, meta) for c in TASK_CHECKS]   # meta = ground truth saved at generation
score  = sum(w * c for w, c in zip(W, checks))   # soft criteria
score  = math.prod(checks)                       # hard constraints

No API calls are needed, GPU compute is optional and only for OCR, and the procedure is fully deterministic. It also decomposes errors: you can see which check a sample failed.

How well does it match humans?

The scorers were validated against human preferences (§3.3). Eight video generation models were evaluated on VBVR-Pro-Bench's 100 tasks with five instances each, producing 4,000 videos. One instance per task was sampled, all model pairs compared, and ten independent human annotations collected for each pair.

The proposed scorer reaches per-vote agreement above 0.60, beating GPT-5.5 at 0.54 and Gemini-3.1-Pro at 0.52. The human agreement ceiling is 0.77, so it lands at roughly 78% of what humans achieve against each other — and it is the cheapest evaluator in the comparison.

Model ranking improved too. Against the earlier VBVR scorer, Spearman's $\rho$ on out-of-domain tasks rose from 0.93 to 1.00 and Pearson's $r$ from 0.92 to 0.95. The final scorer reaches $\rho=0.95$ in-domain, $\rho=1.00$ out-of-domain, with $r\geq0.95$ on both splits (§3.3).

Image, video, or interleaved?

The third pillar is the controlled modality study. Nine open-source generative foundation models were each trained on VBVR-Pro-Dataset for one epoch at $512\times512$. Rank-32 LoRA was applied to all video generators and to image and interleaved models above 10B parameters; smaller ones were fully fine-tuned. Video models trained at 16 fps (§4.1.1).

The headline is clean (§4.1.2). Training improved all nine models, with an average overall gain of +0.290 — +0.401 on in-domain tasks and +0.179 out-of-domain. The best result is the video model VBVR-Pro-Wan2.2-I2V-A14B at 0.670 overall, with the interleaved VBVR-Pro-SenseNova-U1 second at 0.638. Trained open-source models thus overtake strong proprietary systems such as Nano Banana Pro (0.564) and Seedream-5.0-Pro (0.557).

The relationship between modalities is more subtle (§4.1.3).

• Interleaved matches video on familiar tasks. After task-specific training, the strongest interleaved model performs on par with the strongest video model across many in-domain tasks. Once the task structure is learned, a sparse sequence of discrete but task-relevant visual states suffices — and it costs far less to generate and run.
• Video wins when state transitions must be modeled accurately. Video models lead on in-domain transformation tasks and across out-of-domain categories. Dense temporal representation pays off when you must track fine-grained change over time or generalize learned transition dynamics past the training distribution.
• Single-image generation is clearly weaker. One endpoint image cannot express the intermediate states of the solution, so it offers little support for multi-step transitions.

What carries the reasoning is the picture, not the prose

This is the most interesting result in the paper. Interleaved models emit images and text in alternation — so which one is doing the reasoning?

On the training side (Table 5), replacing the intermediate text with semantically uninformative placeholders barely moves the overall score (0.373 → 0.374 and 0.638 → 0.629). But compressing the visual trajectory into a single image costs 0.024 and 0.111 respectively.

The inference-time interventions (Table 6) are blunter still. Against a clean reference of 0.638 overall:

• Rephrase the input text, meaning preserved → 0.640 (essentially unchanged)
• Strip task semantics from the input text → 0.317
• Remove the input image → 0.064
• Flip horizontally or rotate 180°, adjusting direction words to match → 0.613
• Remove the intermediate text → 0.533; contradict it (swap "left" for "right") → 0.530
• Remove the intermediate image → 0.099; corrupt it with strong noise → 0.190

Indifferent to paraphrase, destroyed by removing the picture. Contradicting the intermediate prose is a flesh wound; breaking the intermediate picture is critical. The paper's conclusion is that visual trajectories are not only the stronger training signal but the working representation actually used at inference (§4.2.2). Language plays a supporting role, not the role of primary computational substrate.

Visualizing the diffusion denoising process shows the same thing from another angle. Both video and interleaved models explore several candidate routes in early denoising steps (Multi-Path Exploration) and overlay multiple plausible visual states (Superposition-Based Exploration) before converging. The paper calls this Chain-of-Step (§4.3).

Transfer to external benchmarks was measured too (Table 7). Against its base model Wan2.2-I2V-A14B, VBVR-Pro-Wan2.2-I2V-A14B goes from 10.21 to 38.22 on V-ReasonBench, 24.76 to 48.39 on MME-CoF-Pro, 25.71 to 52.86 on VideoThinkBench-mini, and 0.36 to 12.50 on BabyVision-Gen. Those benchmarks contain real-world scenes and robotic manipulators, a long way from procedurally generated abstractions.

Running RL on verifiable rewards

The last pillar is reinforcement learning (§5). The difficulty is that conventional generative RL rewards pixel fidelity or aesthetics, whereas native visual reasoning must reward satisfying semantic transformations and task constraints.

There is also an exploration problem. Group-relative methods such as Flow-GRPO assume sampled rollouts differ in reward-relevant decisions. But the decisions that matter in visual reasoning are discrete and high-impact — turn left or right in the maze, connect these two objects or those — and ordinary stochastic perturbation tends to change only low-level appearance while leaving the decision intact. Cranking up the noise is not free either: with Flow-SDE, $\eta=0.7$ introduces background artifacts and distorted object boundaries, and $\eta=0.9$ severely corrupts frames.

Hence CPS (Coefficients-Preserving Sampling):

In words, the total amount of noise stays fixed while its makeup shifts from the predicted noise toward fresh randomness. Because the squared cosine and sine coefficients always sum to one, turning up $\eta$ makes the model's decisions more varied without wrecking the picture, and at $\eta=0$ it reduces to deterministic ODE sampling.

$\widehat{x}_{0}$ is the model's predicted clean sample, $\widehat{x}_{1}$ its predicted noise, $\epsilon$ a freshly drawn Gaussian, and $\eta \in [0,1]$ the stochasticity dial. In plain terms: the total amount of noise stays fixed, and only its composition shifts from "predicted noise" toward "fresh randomness." Because $\cos^{2}(\eta\pi/2)+\sin^{2}(\eta\pi/2)=1$, the coefficients always sum to one in square, and $\eta=0$ falls back to deterministic ODE sampling. Turning the dial up therefore diversifies decisions without wrecking the picture.

The experiments start from VBVR-Pro-Wan2.2-TI2V-5B and use the same 50K RL subset spanning 50 in-domain tasks to compare three recipes: SFT, RLVLM (RL with Qwen3.6-27B as judge), and RLVR (RL with the proposed scorer). Algorithm, rollout configuration, hyperparameters, data, and initialization are identical; only the source of the reward differs (§5.2).

Under matched evaluation with CPS at $\eta=0.7$ (Table 8), the baseline scores 0.470, SFT 0.503, RLVLM 0.508, and RLVR 0.548. Out-of-domain, the progression is 0.300 → 0.328 → 0.345 → 0.377, improving on tasks that RL never trained on. Over training, RLVR trends steadily upward while RLVLM stagnates early and dips temporarily around steps 400–500 (§5.3).

The qualitative change matters as much. On a color-sequence completion task, the pre-RL model commits to an answer within the first few denoising steps and never revisits it, while the RLVR model revises its prediction mid-trajectory and lands on the correct color. Both use the same 30-step schedule, so this is not bought with extra inference compute. On a maze, the pre-RL model produces disconnected movements that never form a valid route (score 0.2249) while RLVR follows connected corridors to the target (0.8692). RLVR extends the effective reasoning horizon within a fixed denoising budget and improves temporal consistency across frames (§5.4).

What the paper does not show

For honesty about scope:

• Human-level is still far off. The paper states plainly that even the strongest trained model remains well below human-level performance (§4.1.2). An overall score of 0.670 means 0.670 on a scorer whose maximum is 1.0 — not "solved."
• The scorers are not a general-purpose grader. Each of the 100 benchmark tasks has its own hand-written scorer, viable only because procedural generation hands you ground-truth metadata. It does not lift wholesale onto arbitrary real-world tasks.
• Human agreement is not near-perfect. Above 0.60 per vote is roughly 78% of the 0.77 human ceiling — a relative win, not a high absolute number. The paper notes annotators themselves disagree on close calls.
• The diagnostics are model-specific. Every counterfactual runs on VBVR-Pro-SenseNova-U1, and the RL comparison starts from a single VBVR-Pro-Wan2.2-TI2V-5B checkpoint. How far "visual trajectories are the substrate" generalizes beyond these models is untested here.
• RL coverage is narrow. RL training touches only the 50 in-domain tasks, with evaluation on 100. Out-of-domain gains are observed, but generalization beyond the RL training distribution stays an open question.
• Intermediate text for the interleaved setting was generated with Gemini-3.1-Pro-Preview (§2.2), so its quality is baked into the dataset's assumptions.

How this plays out in practice

Who touches it, and when. Researchers working on image and video generation, plus the ML engineers and QA people who ship generative AI in a product. Three moments matter.

1. Building an evaluation pipeline. The most portable lesson is that synthetic data can emit its own ground truth. VBVR-Pro saves, per instance, the random seed, problem specification, complete solution, and key element attributes as metadata (§2.2). That is what makes rule-based scoring possible instead of outsourcing to a VLM. If you build a synthetic eval set, make the renderer emit a structured representation of the answer alongside every image. Try to grade later from pixels alone and you are back to a VLM judge.

2. Designing a reward function. Before wiring up RLVR, run a reproducibility test: score the same input twice and diff. The 54.6%–92.8% figure in Table 2 is what temperature 0 buys you. A nondeterministic reward puts noise, not task signal, into the gradient. Aggregation is a design decision too: constraints you refuse to violate belong on the multiplicative side, or partial credit will pay for cheating.

3. Choosing sampler settings. The parameters you touch are CPS $\eta$, rollout count, denoising steps, and LoRA rank. The paper's settings: $\eta=0.7$, rank 32, $512\times512$, 16 fps for video. If you are tempted to push exploration with Flow-SDE instead, look at the frames. At $\eta=0.9$ the paper measured corrupted output and falling rewards.

Traps that turn into incidents.

• Never put a VLM judge's score in a doc as an absolute number. A figure that moves up to 0.221 on rerun cannot support a 0.02 gap between model A and model B; next week the conclusion flips. Compare within a single run, ideally with a deterministic scorer.
• Do not grade on pixels. Slight shifts and rendering-style differences will fail correct answers (§3.2). Extract attributes first, then compare.
• Do not treat intermediate visuals as decoration. Table 6 shows 0.638 falling to 0.099 when the intermediate image is removed. "The intermediate images only go to logs, let's drop them" is an optimization that breaks the reasoning itself.
• Always report ID and OOD separately. In-distribution gains (+0.401) and transfer to unseen tasks (+0.179) differ by more than a factor of two. A deck showing only one of them is exaggerating.

How it shows up in an interview or design review. Asked "how would you automatically evaluate generative model outputs?", a clean answer runs: (1) design generation so a structured representation of the answer is saved; (2) match on attributes rather than pixels; (3) multiply hard constraints, weight-and-sum soft criteria; (4) always measure reproducibility on identical inputs. Stopping at "have an LLM grade it" walks straight into the three failure modes this paper documented.

Related reading

• LLM Evaluation — the general limits of LLM-as-a-judge, and how to design around them
• Flow Matching from Scratch — the flow-based generation that CPS and Flow-SDE build on
• Diffusion Models: An Introduction — what the denoising steps behind Chain-of-Step actually are