Paper Walkthrough: SA-MRPO — Stop Studying the Subject You've Already Aced

When you train with several reward objectives at once, gradient budget keeps flowing to objectives that are already solved. SA-MRPO (arXiv:2608.16072) measures how saturated each objective is and discounts its weight accordingly, redirecting optimization toward whatever headroom remains. A walkthrough grounded strictly in the paper.

Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

Studying the subject you already ace, for the same hour every day

You're already scoring 90% in English. You're at 40% in math. If a student kept splitting study time evenly between the two, any decent tutor would step in. That hour of English buys you 90% → 91%; that hour of math might buy you 40% → 50%. The remaining headroom is different.

Almost exactly this happens in language model post-training, and that is where this paper starts. Reasoning models today are asked to be correct, to not run too long, to obey a required format, and — for code — to actually run (§1). The standard recipe collapses all of that into one scalar with a fixed weighted sum before learning from it. The paper identifies two defects in that design: resolution is destroyed, and the weights are blind to the fact that an objective is already solved (§1).

Background: GRPO grades siblings against each other

GRPO (Group Relative Policy Optimization) throws away the learned value function PPO needs. Instead it compares several rollouts (attempts) at the same question and derives an advantage — a signed score for how good that attempt was — from the comparison (§1, §3). Because learning is driven by rank within a sibling group rather than by an absolute score, you never have to train a second model to estimate value. For a query $q_i$, a frozen behavior policy generates $G \geq 2$ responses; the paper uses $G=8$ throughout (§5.1).

With multiple objectives, the standard move is to flatten the reward vector into a scalar first (§3).

Here $r_k^{(i,j)}$ is the score objective $k$ assigned to the $j$-th response, and $w_k$ is the weight a human fixed in advance. Put in words, that is: take the per-subject scores and add them up using a fixed grading scheme. That combined score is then standardized within the group of $G$ siblings born from the same query, which is GRPO's advantage (§3).

$$A_{\mathrm{GRPO}}^{(i,j)} = \frac{r_{\mathrm{sum}}^{(i,j)} - \mathrm{mean}\{r_{\mathrm{sum}}^{(i,1)},\ldots,r_{\mathrm{sum}}^{(i,G)}\}}{\mathrm{std}\{r_{\mathrm{sum}}^{(i,1)},\ldots,r_{\mathrm{sum}}^{(i,G)}\}}$$

What the fraction says, in words: above the sibling average, get praised (positive); below it, get scolded (negative); how far off you are sets the strength. That number then feeds the usual clipped surrogate objective.

Failure mode 1: addition erases resolution

The information is lost the moment you add. Under equal weights, the reward profiles $(1,0)$ and $(0,1)$ produce the same combined score (§4). "Perfect format, wrong answer" and "right answer, broken format" receive identical advantages, because a total of 1 does not remember which subject the 1 came from.

Figure 1 of the paper shows this on a group of $G=4$ rollouts, with a format objective that is already saturated and a correctness objective that is not. Rollouts 2 and 3 have the same combined score but different profiles, so GRPO hands both an advantage of exactly zero (§4). As a training signal, that reads "both average, nothing to say." The prior answer to this failure is GDPO, which normalizes each reward dimension independently before aggregating, preserving what scalarization would have crushed (§2).

Failure mode 2: the weights don't know it's already solved

But the paper argues that fixing 1 leaves 2 untouched (§1). In the same Figure 1, GDPO does tell rollouts 2 and 3 apart — and then hands the larger advantage to rollout 3, the one with zero correctness (§4). Even with the format objective pinned near its ceiling, it keeps the weight $w_k$ that was chosen at the start. So training keeps spending gradient budget on objectives that are already solved, and the ones with real headroom wait their turn (§1). It's the student from the opening, again.

The mechanism: measure saturation, discount by exactly that much

SA-MRPO — Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization — is refreshingly plain. Standardize each objective independently within its group, then discount its contribution by how much of that objective is already solved (§4.1).

To capture the current state of objective $k$, take its mean reward $\bar{r}^{(k)}$ over the entire current batch. Batch-level rather than group-level is the point: it answers "how well is this objective doing overall right now?" Combined with the attainable lower and upper bounds $r_{\min}^{(k)}$ and $r_{\max}^{(k)}$, that gives the saturation ratio (§4.1).

That is, in words, what fraction of the available points you have already banked. Small means lots of headroom left; close to 1 means pinned against the ceiling. For a binary 0/1 reward it is literally the success rate.