Paper Walkthrough: Annotations as Rollouts — Dropping the Ground Truth Into the Group as a Ninth Answer

RL post-training for video MLLMs starves because sampled rollouts almost never contain the right answer. OraRL adds the annotation itself to the group as an extra rollout, then repairs the advantage inversion that naive mixing causes.

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

The grader who never shows the answer key

Picture a grader who hands the same problem to a student (the model) eight times, scores all eight answer sheets, and returns them. The grader is holding the answer key the whole time — and never shows it. The student has to learn from nothing but "which of these eight is least bad."

For a multiple-choice question that still works, because the right answer shows up somewhere in the eight. But when the answer is "12.4 to 18.9 seconds into the video," all eight sheets miss. A batch containing nothing close to the right answer says very little about which direction to improve in.

This paper (arXiv:2608.20492) makes a one-line pivot: take the answer the grader is already holding, and drop it into the batch as a ninth answer sheet. The authors call the principle annotation-as-rollout, and the training recipe built around it OraRL.

GRPO, from the ground up

The reinforcement learning here sits in the PPO lineage covered in instruction tuning and RLHF: GRPO (Group Relative Policy Optimization). Three steps (§3.1).

1. For one input (video plus instruction), sample $n$ outputs from the model. Each one is a rollout
2. Score each rollout against the annotation to get a reward $r_i$ (say, how much the predicted interval overlaps the annotated one)
3. Judge each rollout by whether it beat the group average

Step 3 is the trick that lets GRPO skip training a separate value function (critic).

$A_i$ is the advantage: the coefficient deciding whether that rollout gets pushed up or down in probability. $\mu_{\mathrm{grp}}$ is the mean reward inside the group, $\sigma_{\mathrm{grp}}$ its spread, and $\epsilon$ a small constant guarding against division by zero. Written out in words: "solve the same problem eight times, and praise only the answers that beat the average." Positive $A_i$ means push up, negative means push down.

Video tasks rarely produce a positive anchor

Now the video-specific problem. Unified video perception (see the video understanding landscape) deals in continuous, structured answers: time intervals, boxes, segmentation masks, tracking trajectories. Unlike multiple choice, sampling almost never lands on one exactly.

As the paper puts it, in existing methods the annotation serves only as a scoring reference, so many groups get updated with no reliable positive anchor (§1). Measured: on temporal grounding, sparse rewards leave 17.5% of groups without a single positive rollout (§4.10).

The intuition "just let it reason longer with chain-of-thought" does not survive contact with the numbers. GRPO with CoT averages 58.5 across three temporal-grounding datasets versus 58.7 without it — statistically a wash — while step time climbs from 93.9 s to 135.6 s (+44.4%). Worse, even before RL, adding CoT drops the raw backbone average by 3.9 points (§4.7). Thinking longer does not make you hit a target you were never going to hit.

Making the annotation the ninth rollout

The core of OraRL is almost anticlimactically simple (§3.2). Define a transform $T_{\mathrm{task}}$ that rewrites the annotation $y$ into the model's own response format, then append it to the group as an oracle rollout.

$$o_{\mathrm{gt}}=T_{\mathrm{task}}(y),\quad \mathcal{O}_{\mathrm{aug}}=\mathcal{O}_{\mathrm{op}}\cup\{o_{\mathrm{gt}}\},\quad |\mathcal{O}_{\mathrm{aug}}|=n+1$$

$o_{\mathrm{gt}}$ is the ground truth written out as if the model had produced it, $\mathcal{O}_{\mathrm{op}}$ is the model's own $n$ rollouts, and $\mathcal{O}_{\mathrm{aug}}$ is the combined set of $n+1$. Put in words, this is an addition, not a replacement: all eight on-policy rollouts survive, so exploration is untouched and only the positive anchor is added. What goes inside $o_{\mathrm{gt}}$ varies by task — an answer choice, a time interval, a box, a box trajectory — but one update rule covers all of them (§3.3). Unlike knowledge distillation, which imitates a teacher model's output, this oracle is the annotation that shipped with the dataset, so no teacher is required (§2).