Paper Walkthrough: SemComp-Bench — Asking Video Generators 'So Did It Actually Get Done?'

SemComp-Bench moves video-generation evaluation from 'does it look good' to 'did it reach the outcome I asked for'. A walkthrough of the task definition, the four-stage data pipeline, the AND-vs-average scoring, and the measured numbers — all from the paper itself.

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

Did the Banknote Actually Become a Turtle?

There is a banknote on the table. The instruction is one line: "Fold the banknote into a turtle." Hand that to a video generation model and you will usually get something plausible back. Fingers move, paper bends, the lighting behaves. But freeze the last second and look: there is no turtle. Or there is a turtle — except the thing on the table is no longer your banknote, it is some other piece of origami that wandered in from somewhere else.

SemComp-Bench exists to measure exactly that "so did it actually get done?" question. The authors define success as satisfying two conditions at once: the video must achieve the intended outcome, and that outcome must be semantically grounded in the reference image. They call the resulting problem Semantic Task Completion Video Generation (§1).

The banknote is the paper's own example. Given the banknote image and the instruction to fold it into a turtle, a successful video must visibly present the referenced banknote as turtle-shaped origami. The intermediate folding steps are unnecessary — but if the result is not grounded in the original banknote and has been swapped for an unrelated turtle, it fails (§1).

Lining this up against what existing benchmarks measure makes the positioning clear. Per the paper's related work (§2.1), prior evaluation axes center on visual fidelity, temporal coherence, and prompt adherence, joined more recently by subject identity, intended motion, physical plausibility, and commonsense consistency. Every one of those is an axis of video quality. What was missing, the authors argue, is a framework that asks whether the video reached an outcome jointly defined by an instruction and a reference context, while preserving task-relevant grounding.

Deciding What Not to Measure

The sharpest design decision here is not what the benchmark scores — it is what it deliberately refuses to score. Evaluation looks only at the generated outcome, and requires neither (a) a complete sequence of intermediate task steps, nor (b) conventional appearance consistency with the reference image (Abstract, §1).

Why grant those exemptions? Filming the entire act of folding a banknote does not physically fit into a few seconds of video. Penalize missing steps and you throw out a pile of videos that got the job done. Appearance consistency has a worse problem: turning a banknote into a turtle is a change in appearance. Demand pixel-level closeness and you have built a metric that rewards not changing anything — evaluation standing on its head.

So the paper replaces appearance consistency with semantic grounding: only the high-level, task-relevant semantics need to correspond to the reference, while unrelated attributes are free to change (§1). The origami turtle must be the one folded from that banknote. Whether the creases and the lighting shifted is nobody's business.

The task pool is concrete too. It covers six domains of everyday hands-on work — Food and Cooking, Beauty and Fashion, Crafts and DIY, Gardening and Pets, Sports and Fitness, and Arts and Precision — split into 21 fine-grained categories (§3.1). Ingredients becoming a dish, a damaged object being restored, a blueprint becoming a physical build: every one is a change you can verify with your eyes. Videos whose success you can only judge by listening to narration are filtered out from the start.

Throwing Away the One-Dimensional Ruler

Conventional consistency metrics, stripped down, reduce to "how closely do these two vectors point the same way?" Convert to features, take a dot product or a cosine, call bigger more similar. It is a one-dimensional dial. Turn it by hand first:

What the dial reveals is that it is monotone. Farther apart, lower; closer, higher. But "fold the banknote into a turtle" is an operation that moves appearance a long way while holding the semantic correspondence fixed. Raising or lowering that similarity never answers the questions you actually care about: is it a turtle, and is that turtle made of your banknote? SemComp-Bench drops continuous similarity in favor of binary questions precisely to escape the monotone dial.

What does binarizing buy you? The paper's answer: requiring the evaluator to support each answer with visual evidence yields focused, interpretable judgments and enables criterion-level failure diagnosis (§1, §4). What you take home is no longer "a score" but "which condition it failed on."

How the Data Was Built — A Four-Stage Pipeline

The underlying SemComp-Data consists of triplets — a reference image $r_i$, an aligned instruction pair $\mathcal{I}_i$ (one brief, one detailed), and an outcome-centric clip $o_i$ — and crucially, all of them are cut from the same single full-context source video in Koala-36M (§3). Because reference and outcome come from one real episode of work, the pipeline structurally guarantees both that the task is achievable and that the attributes genuinely correspond.