Paper Explained: RewardVerse — A Video Reward Model That Writes the Rubric Before It Scores (RGPO)
Video reward models tend to squeeze their scores into a narrow band, and those scores move when the wording changes. The paper calls this scalar drift. RewardVerse first generates an evaluation rubric, then reads a continuous score for each item from token probabilities. Its two-stage RGPO training uses only 30 preference pairs per dimension.
Key points
- RewardVerse is a video reward model that inserts a dynamic rubric between the evaluation query and the scorer.
- Existing video RMs map quality to one number with no explicit criteria, which causes scalar drift.
- Scalar drift is defined by two symptoms: score-range collapse and high variance under context shift.
- The training algorithm RGPO has two stages: it warms up the scorer with self-evolving seed rubrics, then trains the rubric generator while keeping the scorer aligned.
- In the pilot study, with soft-logits fixed, V2→V3 raised the standard deviation from 0.258 to 0.471, a rise of 82.6%, but still deviated from human ratings.
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
Primary source — what this article is built on
undefined2026-09-19→undefined2026-09-29same month
RewardVerse: Rubric-Guided Policy Optimization for Video Reward ModelingZhenchen Tang, Yang Li, Songlin Yang et al. · 2026-09-19 · v2arXiv:2609.22947Paper page·PDFundefined
Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
What the paper claims
The paper is "RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling" (arXiv:2609.22947, September 2026, v2).
Here is the abstract in brief. Training video generators with reinforcement learning (RL) depends on a robust reward model (RM). Existing video RMs, though, take something subjective and multi-faceted (how good a video is) and turn it straight into one number, with no explicit criteria. As a result, the scoring scale collapses or shifts from one prompt to the next. The authors call this scalar drift, and it makes the reward unreliable for RL. Borrowing from professional human annotation practice, they propose RewardVerse, which inserts a dynamic rubric between the evaluation query and the scorer. The model writes explicit criteria first and then scores against them, and the criteria act as an anchor against drift. The training algorithm, RGPO, runs in two stages. It first warms up the scorer with self-evolving seed rubrics. It then trains the rubric generator to write criteria tailored to each query while keeping the scorer aligned with human ratings. On the 16-dimension EvalVerse benchmark and on external data, the authors report state-of-the-art results for both pointwise and pairwise evaluation.
An analogy: a judge without a scoresheet
Ask a cooking judge "out of 5, what does this dish get?" and you'll usually hear "about a 4." If every plate lands between 3.8 and 4.3, the scores barely tell the dishes apart. Rephrase the question or change the serving order, and the numbers wobble. A professional judge works differently. They first write a scoresheet ("doneness," "seasoning," and so on), score each line, and then take a weighted total. RewardVerse gives one model both jobs, writing the scoresheet and scoring against it, and trains both with RL.
Scalar drift and a 2×2 pilot study (§3)
The paper defines scalar drift by two symptoms (§3.1). The first is score-range collapse: scores bunch up in a safe, high band. The gap between good and bad videos shrinks, and RL is left with nearly flat gradients. The second is high variance under context shift: paraphrasing the instruction or reordering the inputs changes the score the same video gets.
In §3.2 the authors test "rubric or no rubric" against "how the score is read out." The setup is Qwen2.5-VL-7B on 240 human-labelled videos. The two readout options are asking the model to write the number as text, or reading it via soft-logits. With soft-logits, you look at how likely the tokens "1" through "5" are at the answer position and use the expected value as the score. A model torn between 3 and 4 then gives something near 3.5.
Adding a rubric widened the spread of scores. With soft-logits held fixed (V2→V3), the standard deviation went from 0.258 to 0.471, a rise of 82.6%. When the model wrote its number as text, though, about two-thirds of pairs still tied even with a rubric. Only V3, which combines a rubric with soft-logits, spread the distribution out. Even V3 still deviated from human ratings, and that gap is why training is needed.
Play with the sharpness of the "1"–"5" distribution below. When it is sharp, the expected value snaps to a whole number; when it is flat, it lands in between.
Comments
Sign in to comment