Paper Explained: VDiff-Bench — Even Frontier MLLMs Miss the Low-Level Changes in a Spot-the-Difference Puzzle

VDiff-Bench asks a multimodal LLM what changed between two near-identical images, as a four-way multiple choice. We walk through how its 1,756 questions across 10 change categories were built, what 11 models scored, why several of them answer 'no difference' when noise or texture changed, and what this means if you use an MLLM to grade image edits.

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

In one sentence

Multimodal LLMs (MLLMs) are strong at answering questions about a single image, yet surprisingly weak at a task any child can do: look at two near-identical images and say what changed. The paper "VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification" (arXiv:2609.06245), by Yixin Wan, Tianle Zheng, and Kai-Wei Chang at UCLA, builds a four-way multiple-choice benchmark for exactly this skill, runs 11 current models through it, and finds that several of them can spot semantic changes but are nearly blind to low-level ones such as noise and texture.

Here is the abstract in brief. VDiff-Bench contains 1,756 four-way questions covering 10 change categories: position, motion, regional color, whole-image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question pairs two images with four options: the true difference, two hard-negative descriptions curated to sit close to the ground truth, and a "no difference" distractor. Across 11 open- and closed-source MLLMs, fine-grained visual comparison remains brittle: three 7–8B open-source models score 52.5–70.6% on semantic changes but only 8.7–33.3% on low-level changes, falsely assuming nothing changed. And despite strong results from other commercial models, Grok 4.3 shows a striking drop on noise and texture differences, falling well behind large open-source models like Kimi K2.5 and K3.

The metaphor: handing a spot-the-difference puzzle to an AI

Think of the puzzle at the back of a children's activity book: "find what's different between the top and bottom picture." The paper poses precisely this to MLLMs (§1 describes the task as one that "commonly exists in children's playbooks"). And models that look extremely capable will, when the whole photo has a faint film of grain added or a face's skin has been smoothed to a plastic finish, confidently declare that the two images are identical.

The authors name the capability Image Difference Identification (IDI) and break it into three requirements (§1): comparative perception across two views, sensitivity to small localized or low-level differences, and restraint against hallucinating changes that are not there.

Why a new benchmark was needed

Describing the difference between two images is an old task; Spot-the-Diff (surveillance footage) and CLEVR-Change (rendered scenes) are well known. The paper points to two weaknesses in existing benchmarks (§1, §2.2).

1. The data is too easy. A current model like Gemini 3.5 not only verbalizes every difference in the rendered scenes, it lists camera-angle changes the benchmark's own ground truth missed.
2. The scoring is unreliable. Because models write free-form captions, you either compare against reference captions with BLEU-4 or ROUGE-L (sensitive to paraphrase, unable to cleanly separate omitted, reversed, or unsupported claims) or you use an MLLM as judge (with a large literature on judge bias and fragility).

So the paper reformulates the task as a four-way discrimination problem: can the model tell the change that actually happened from plausible changes that did not, scored by exact answer-key matching (§3.6)? The judge model disappears. The trade-off, which the authors state openly, is that the benchmark no longer measures completeness, i.e. whether a model can list every difference (§2.2).

§2.3 explains why this matters for image generation. Evaluating an image-editing model means checking both that the requested change happened and that nothing else (e.g. the background) changed, and MLLM judges are now widely used for this. Yet those judges routinely fail to describe fine-grained edit-induced differences and frequently hallucinate changes. IDI accuracy is therefore a foundation for editing evaluation and for reward models built on it.

The mechanism: one question and the scoring rule

Each item is an ordered image pair $(I_A, I_B)$ plus four textual options (§3.1): one reference-supported true difference, two candidate alternatives intended to be false (hard negatives), and one fixed option claiming the images are completely identical. Every pair contains at least one real change, so "no difference" is always a distractor. The model returns a single label A–D, and accuracy is:

Here $N$ is the number of questions (1,756), $\hat{y}_i$ is the label the model chose, $y_i$ is the correct label, and $\mathbb{1}[\cdot]$ is an indicator that returns 1 when the condition holds and 0 otherwise. In words: the fraction of questions where the chosen letter matches the key. Deliberately plain. All the difficulty is pushed into how the questions are built.

Wrong answers come in two flavors (§3.1): picking "no difference" is logged as a missed-change selection, picking a hard negative as a competing-change selection. That distinction drives the error analysis later.

The 10 categories and their question counts (Table 1):

The paper groups whole-image color, noise/resolution, texture, and illumination as low-level changes and the remaining six as semantic changes (§3.5). The 1,756 questions come from 1,543 distinct pairs; a pair with several annotated differences yields several questions.

You can feel how subtle low-level changes are with a familiar example. The figure below shows lossy-compression degradation: put two versions a few quality points apart side by side and it is genuinely hard to say in words what changed.