Paper Walkthrough: On-Policy Reverse Distillation — Borrowing a Weak Teacher's Direction, Not Its Answers

OPRD takes the policy change a weak teacher acquired during post-training and uses it only as a direction to amplify the student's own policy gradient — never as an optimization target. A ground-up walkthrough.

Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

Can a strong student outgrow a weak teacher?

The paper we are reading is "Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation" (Youngrok Park et al., arXiv:2609.08798, published 2026-09-08, 38 pages).

Here is what the abstract claims. Weak-to-strong generalization asks whether a stronger model can learn from a weaker supervisor and then surpass it. That question matters commercially because redoing frontier-scale post-training from scratch — for each new model generation, or every time you want to fold several domains into one model — is prohibitively expensive. Conventional distillation, however, treats the weak teacher as the optimization target itself, which risks imposing the teacher's capacity ceiling on the student. The paper's method, On-Policy Reverse Distillation (OPRD), evaluates how far the teacher's policy has shifted relative to its own reference policy — measured on the student's own rollouts — and amplifies only the component of the student's verifier-driven policy gradient that lies along that direction. Because it rescales only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning past the teacher. In both successive model transfer and multi-teacher distillation, the authors report higher performance with fewer student updates than existing RL and distillation approaches.

An analogy: borrow the improvement, not the moves

Imagine you are a strong amateur chess player, coached by someone who was a mediocre club player last year but has improved a lot since.

If you memorize your coach's current moves, your ceiling becomes their ceiling. That is conventional distillation. But if you extract only what they changed over the year — stopped forcing bad attacks in the opening, started valuing tempo over material in the endgame — that is a more general lesson, and it does not cap you.

OPRD does the latter, with one extra twist: it does not make that direction a target either. Of the improvements you derive from your own game results, it simply pushes harder on the ones pointing the same way your coach moved. That line — never a target, only an accelerant — is the heart of the design.

Background: RLVR, and its teacher-driven cousin OPD

Modern reasoning post-training centers on RLVR (Reinforcement Learning with Verifiable Rewards) (§2.1): optimize the policy using only rewards a program can check, such as an exact-answer match or a passing test suite. The student policy $\pi_\theta$ samples a response and visits prefixes $s_t=(x, y_{<t})$; the per-token policy gradient is Eq. (2.1) in the paper:

In words, this is the "if this token was good, raise its logit" direction, written as a vector the size of the vocabulary: the advantage $A_t$ says how good the token turned out to be, and the gradient says which way to move the logits.

$A_t$ is the advantage — how good that token turned out to be — and $\mathbf{z}_t$ is the next-token logit vector (the raw numbers before softmax). So Eq. (1) is simply "if this token was good, here is the direction that raises its logit," written as a vector the size of the vocabulary.

On-Policy Distillation (OPD), by contrast, samples responses from the student, queries the teacher at each prefix the student actually visits, and matches the student to that distribution. This fixes the mismatch between training-time and inference-time prefixes, which is why it became a standard step. But the optimum of that objective is the teacher's policy itself (Eq. 2.3). A weak teacher's final policy blends three things: what it genuinely learned during post-training, preferences inherited from its reference policy, and behavior forced by its limited capacity. OPD transfers all three. And adding RL on top does not resolve the tension while teacher matching remains a separate objective that can compete with reward maximization (§1).

The mechanism: keep the direction, throw away the magnitude

OPRD uses not the teacher's final policy but the change post-training produced (§2.2). Let $\pi_T$ be the frozen RL-trained teacher and $\pi_T^{\rm ref}$ its frozen pre-RL reference policy. At a student-visited prefix $s_t$, take the difference of their next-token logits and mean-center it (subtract the mean from every component, written $\mathcal{C}$); the result is the teacher policy shift $\bm{\Delta}_t$. Shifting all logits by a constant leaves the post-softmax distribution unchanged, so removing that common offset discards nothing meaningful.

The crucial move is that OPRD keeps only the unit direction, discarding the magnitude:

In words, $\mathbf{d}_t$ keeps only which way the teacher's policy shifted and discards how far it shifted, so the shift becomes a pure unit-length direction.

Eq. (2) says: retain which way the teacher moved, not how hard. The stated reason is clean — the shift was learned inside the weak teacher's policy class, so it carries no guarantee of improving verifier reward for a stronger student (§2.2). In practice the authors also apply a top-10 truncation under the student policy, concentrating the correction on the vocabulary region the student actually puts probability on.

The sign of a dot product decides follow vs. defy