Paper Walkthrough: Annotations as Rollouts — Dropping the Ground Truth Into the Group as a Ninth Answer

RL post-training for video MLLMs starves because sampled rollouts almost never contain the right answer. OraRL adds the annotation itself to the group as an extra rollout, then repairs the advantage inversion that naive mixing causes.

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

The grader who never shows the answer key

Picture a grader who hands the same problem to a student (the model) eight times, scores all eight answer sheets, and returns them. The grader is holding the answer key the whole time — and never shows it. The student has to learn from nothing but "which of these eight is least bad."

For a multiple-choice question that still works, because the right answer shows up somewhere in the eight. But when the answer is "12.4 to 18.9 seconds into the video," all eight sheets miss. A batch containing nothing close to the right answer says very little about which direction to improve in.

This paper (arXiv:2608.20492) makes a one-line pivot: take the answer the grader is already holding, and drop it into the batch as a ninth answer sheet. The authors call the principle annotation-as-rollout, and the training recipe built around it OraRL.

GRPO, from the ground up

The reinforcement learning here sits in the PPO lineage covered in instruction tuning and RLHF: GRPO (Group Relative Policy Optimization). Three steps (§3.1).

1. For one input (video plus instruction), sample $n$ outputs from the model. Each one is a rollout
2. Score each rollout against the annotation to get a reward $r_i$ (say, how much the predicted interval overlaps the annotated one)
3. Judge each rollout by whether it beat the group average

Step 3 is the trick that lets GRPO skip training a separate value function (critic).

$A_i$ is the advantage: the coefficient deciding whether that rollout gets pushed up or down in probability. $\mu_{\mathrm{grp}}$ is the mean reward inside the group, $\sigma_{\mathrm{grp}}$ its spread, and $\epsilon$ a small constant guarding against division by zero. Written out in words: "solve the same problem eight times, and praise only the answers that beat the average." Positive $A_i$ means push up, negative means push down.

Video tasks rarely produce a positive anchor

Now the video-specific problem. Unified video perception (see the video understanding landscape) deals in continuous, structured answers: time intervals, boxes, segmentation masks, tracking trajectories. Unlike multiple choice, sampling almost never lands on one exactly.

As the paper puts it, in existing methods the annotation serves only as a scoring reference, so many groups get updated with no reliable positive anchor (§1). Measured: on temporal grounding, sparse rewards leave 17.5% of groups without a single positive rollout (§4.10).

The intuition "just let it reason longer with chain-of-thought" does not survive contact with the numbers. GRPO with CoT averages 58.5 across three temporal-grounding datasets versus 58.7 without it — statistically a wash — while step time climbs from 93.9 s to 135.6 s (+44.4%). Worse, even before RL, adding CoT drops the raw backbone average by 3.9 points (§4.7). Thinking longer does not make you hit a target you were never going to hit.

Making the annotation the ninth rollout

The core of OraRL is almost anticlimactically simple (§3.2). Define a transform $T_{\mathrm{task}}$ that rewrites the annotation $y$ into the model's own response format, then append it to the group as an oracle rollout.

$$o_{\mathrm{gt}}=T_{\mathrm{task}}(y),\quad \mathcal{O}_{\mathrm{aug}}=\mathcal{O}_{\mathrm{op}}\cup\{o_{\mathrm{gt}}\},\quad |\mathcal{O}_{\mathrm{aug}}|=n+1$$

$o_{\mathrm{gt}}$ is the ground truth written out as if the model had produced it, $\mathcal{O}_{\mathrm{op}}$ is the model's own $n$ rollouts, and $\mathcal{O}_{\mathrm{aug}}$ is the combined set of $n+1$. Put in words, this is an addition, not a replacement: all eight on-policy rollouts survive, so exploration is untouched and only the positive anchor is added. What goes inside $o_{\mathrm{gt}}$ varies by task — an answer choice, a time interval, a box, a box trajectory — but one update rule covers all of them (§3.3). Unlike knowledge distillation, which imitates a teacher model's output, this oracle is the annotation that shipped with the dataset, so no teacher is required (§2).

Mix it in naively and it breaks: advantage inversion

Normalize all nine together, though, and performance drops below plain GRPO. Under the shared three-task protocol, GRPO's average of 60.3 falls to 55.4 with naive ground-truth injection — tracking alone loses 11.9 points (§4.9). The cause is grade-school arithmetic.

$\mu_{\mathrm{op}}$ is the mean over the model's eight rollouts alone and $r_{\mathrm{gt}}$ is the oracle's reward. Put in words: a high-scoring oracle lifts the baseline, by exactly $(r_{\mathrm{gt}}-\mu_{\mathrm{op}})/(n+1)$. Any rollout whose reward lands in that band was being praised under GRPO and is now being suppressed — the failure the authors name advantage inversion. The band's width scales with the oracle-policy gap, so the worse the model, the more damage it takes, and the oracle also inflates the standard deviation, shrinking whatever advantages do survive.

Pooled over all 92,024 rollouts from 11,503 groups (§4.10):

How OraRL fixes it: exclude from the baseline, route the gap elsewhere

The answer is to exclude the oracle from the baseline while keeping it as an optimization target (§3.2).

1. Compute the baseline from the model's eight rollouts only. Set $A_{i}^{(0)}=r_{i}-\mu_{\mathrm{op}}$, with no division by the variance. "Above average yet negative" is now impossible by construction.

2. Recover the gap information from the change in spread.

$\sigma_{\mathrm{op}}$ is the spread of the eight, $\sigma_{\mathrm{aug}}$ the spread with the oracle included. Stated in words, $g_q$ measures how far the policy still is from the oracle. The fourth root damps it and the clip to $[1,4]$ stops runaway amplification. It is applied only to above-average rollouts, and the mean is subtracted again afterwards so the group still sums to zero.

3. Give the oracle its own, deliberately modest advantage.

$$w_{q}=\left[\operatorname{clip}\!\left(\frac{r_{\mathrm{gt}}-\mu_{\mathrm{op}}}{r_{\mathrm{gt}}+\epsilon},0,1\right)\right]^{2},\qquad A_{\mathrm{gt}}=\min\!\left(2w_{q},\;\operatorname{clip}\!\left(1.2\,A^{+}_{\max},\,0.05,\,1\right)\right)$$

Put another way, $w_q$ is a 0-to-1 weight for "how much headroom is left toward the oracle," and it decays to zero on its own once the policy mean catches up (the square sharpens that decay). $A^{+}_{\max}$ is the strongest positive on-policy advantage, and capping at 1.2 times it means the oracle can never dominate the group. When no positive rollout exists at all, a thin 0.05 bootstrap signal remains.

# OraRL advantages (the essentials, numpy pseudocode)
mu_op = r_op.mean()                                    # oracle excluded
A0    = r_op - mu_op                                   # no variance normalization
g     = np.clip((np.append(r_op, r_gt).std() / (r_op.std() + eps))**0.25, 1, 4)
U     = np.where(A0 > 0, g * A0, A0)                   # amplify the top half only
A_op  = U - U.mean()                                   # re-center to zero
w     = np.clip((r_gt - mu_op) / (r_gt + eps), 0, 1)**2
A_gt  = min(2*w, np.clip(1.2*max(0, A_op.max()), 0.05, 1))

Throw half away and still learn: sign-balanced pruning

Back-propagating through all nine rollouts is expensive, so gradients flow through only $K=\lfloor n(1-\kappa)\rfloor$ of them. At $n=8,\ \kappa=0.5$ that means nine down to four: the oracle, one positive rollout, and two negative ones (§4.1). The point is the sign balance — selecting by magnitude alone, as CPPO does, can retain a single sign and push every surviving rollout the same way. Dropping the sign quota costs 0.6 points on average.

Because the selection is biased by construction, the retained mean and scale are skewed too. Hence moment correction: re-center to zero, then rescale by a factor (clipped to 0.25–1) that matches the pre-pruning RMS. Without it, pruning costs 1.1 points; with it, the loss is held to 0.4 points in exchange for a 1.48× speedup (§4.11).

How far do the results actually go

Headline numbers, all generated answer-only with no CoT decoding (§4.2):

Line up the training methods from the same SFT initialization, data, and budget (§4.8) and the on-policy family clusters within 0.4 points — continued SFT 58.7, GRPO 60.3, Dr. GRPO 60.7, CPPO 60.5, GDPO 60.4 — while OraRL pulls away to 62.7, consistent with the paper's claim that the gain comes from adding the oracle, not from a different advantage estimator. It beats its backbone by more than 8 points at every size from 0.8B to 9B, and its margin over GRPO widens rather than narrows as prompts scale from 6.4k to 100k (§4.3, §4.4). At inference the median response is 13.5 tokens against the backbone's 808.5 reasoning tokens, pulling P90 latency from 62.67 s down to 25.15 s (§4.12).

Limits, and what has not been tested

The paper's own limitations section (§5) is refreshingly blunt. Everything rests on the assumption that an annotation can be serialized into one valid rollout and scored by a scalar reward; behavior under ambiguous, partial, or noisy supervision, and with learned oracles, has not been evaluated. And on spatial intelligence, the MMSI-Bench / MindCube-Tiny average of 47.7 sits below Grok-4's 50.7 and GPT-5's 49.1 (§4.2).

The most revealing result is the re-annotated ReVSI benchmark (Appendix F). Moving from VSI-Bench to ReVSI costs Video-ORA 14.9 points, against only 6.3 for its backbone. Broken down: on relative-direction questions it scores 91.5% forward but 8.3% backward. Relative to the backbone, forward accuracy rose from 77.4% to 85.8% while backward accuracy fell from 18.4% to 9.1%. The training data's templates were forward-only — which is annotation-as-rollout behaving exactly as advertised: a correct oracle buys you nothing in situations the annotations never covered.

Reading it for practice: this works where the ground truth is structured data and the reward can be computed from it mechanically. For summarization or dialogue, where no single answer is correct, the premise of collapsing the oracle into one rollout simply does not hold.

Using this on the job

Who, and when. A post-training engineer who already has an SFT'd video or image VLM and wants to improve tasks whose ground truth lives in the dataset as structured data — time intervals, boxes, masks, trajectories.

Knobs you actually touch. The implementation is veRL, with vLLM for rollout generation and FSDP for training. The settings that matter: n (rollouts per prompt = 8), κ (pruning ratio = 0.5), temperature = 1.0 / top_p = 0.85, kl_coef = 0, PPO clip 0.2, learning rate 2e-6 on a constant schedule, response cap 2048 (Tab. 17). Video is sampled at 2 fps up to 128 frames; tracking alone uses 32 uniform frames (Appendix B).

Failure modes worth knowing before you hit them.

• Do not add the oracle and then normalize over the whole group. That is advantage inversion. In your logs it shows up as "total reward is climbing but one task keeps getting worse" (tracking, −11.9 points). There is no off-the-shelf metric for it: instrument it yourself, per group, as a flip rate — what fraction of rollouts that GRPO would have scored positive went negative.
• If you prune, restore the mean and the scale. Skipping moment correction costs 1.1 points. A good share of "I made it faster and accuracy dropped" lives here.
• Close the reward loopholes in format validation first. Exactly one <answer> block, no leaked <think> tags, zero reward for malformed output. Segmentation adds explicit ceilings — score capped at 0.1 when no positive point falls inside the mask, and at 0.2 when the box IoU is under 0.1 (Appendix C).
• Curate the RL set by difficulty. The paper runs inference with the SFT checkpoint, scores each candidate, and caps examples scoring 0.80 or above at 15% of each subset (Appendix A). Feed it only easy prompts and $w_q$ pins to zero, killing the oracle signal entirely.
• Deduplicate videos between train and eval. The paper detects shared videos and removes the training-side examples. Skip it and every number in your score table is meaningless.

The question you will get in a design review. "What breaks when you drop one worked example into the group?" Write $\mu_{\mathrm{aug}}=\mu_{\mathrm{op}}+(r_{\mathrm{gt}}-\mu_{\mathrm{op}})/(n+1)$, then say that rollouts landing in the band of width $(r_{\mathrm{gt}}-\mu_{\mathrm{op}})/(n+1)$ have their sign flipped, and that the band widens the worse the model is. That is the whole answer.