Paper Walkthrough: No Gold Answers, No Stronger Teacher — How u-OPSD Distills From Its Own Majority Vote

u-OPSD (arXiv:2608.06296) replaces the gold solution in on-policy self-distillation with the model's own majority vote, then corrects only the rollouts that disagree with it. A from-scratch walkthrough grounded solely in the paper.

On-Policy Self-Distillation without Any Supervision

A study room where someone took the answer key away

Picture two students in a study room who happen to be the same person. One of them — the teacher — has the worked solution in hand and solves the problem while looking at it. The other — the student — sees only the problem statement. Now force the student to copy, token by token, what the teacher believes the next move should be. That is on-policy self-distillation (OPSD). Teacher and student share weights, which is why it is called "self" distillation. But what makes the teacher smarter is the worked solution, and that came from outside.

The paper pushes on exactly that point: does the teacher in on-policy self-distillation actually need a ground-truth solution, or can a model build its own privileged context and perform genuine self-distillation (§1)? The answer is u-OPSD. In place of the gold solution it drops in the majority vote over eight of the model's own attempts at the same problem. All it consumes is the problem statement — no gold labels, no environmental feedback, no larger teacher model.

The map: what each recipe was borrowing from outside

GRPO's weakness is that its signal is sparse: a rollout thousands of tokens long receives one bit of information. OPD and OPSD made it dense, telling the model at every single token what the distribution should have been. But as the paper points out, OPSD is self-distillation only in the sense that the parameters are shared — the information that makes the teacher capable still arrives from outside (§1). That is what capped these methods at labeled problems.

The core idea: one sample is unreliable, agreement among eight is a signal

The observation is simple: "although an individual rollout may be unreliable, agreement among multiple independently sampled rollouts provides an endogenous confidence signal" (§1). Wrong answers scatter. A correct chain of reasoning tends to converge on one value, while mistakes land wherever the model happened to slip, so if six of eight rollouts land on the same answer, that answer is worth more than any single one of them.

There is a second payoff. The rollouts that fall outside the majority are, by construction, a list of the places worth fixing. Distillation is applied only there. In the paper's words, this lets the model "correct itself precisely where it is confidently wrong" (Abstract).

Three steps: sample, vote, distill

Algorithm 1 lays out the per-prompt procedure in three stages (§3.2).

(1) Sample. For each problem $x$, draw $G$ rollouts at the training temperature from $\bar\pi$, a stop-gradient copy of the current policy. Pull the final answer $a^{(g)}$ out of \boxed{...} with normalization and canonicalization. Anything unparsable — usually a truncation — is marked invalid.

(2) Vote. The plurality among the valid answers becomes the pseudo-answer $\tilde a(x)$, with ties broken uniformly at random. That splits the rollouts into an agreeing set $\mathcal{Y}^+_x$ and a disagreeing set $\mathcal{Y}^-_x$. Invalid rollouts join neither, because a generation that was cut off is evidence of neither a correct nor an incorrect belief. Confidence in the vote is measured by the self-consistency score:

$c(x)$ is the self-consistency score, $G$ the number of rollouts, $a^{(g)}$ the answer the $g$-th rollout ended up with, $\tilde a(x)$ the answer that won the vote, and $\mathbb{1}[\cdot]$ a counter that returns 1 when the condition inside holds and 0 otherwise. Put in words, it says: count how many of the eight attempts came back with the same answer as the winner, then divide by eight. Equation (1) is a show of hands written as a fraction, and nothing more. One detail matters more than it looks: the denominator is $G$, the total drawn, not the number of valid ones. A prompt that truncates often loses confidence for that reason alone and gets filtered out automatically (§3.2).

(3) Distill. Only prompts whose $c(x)$ reaches the threshold $\tau$ are used. One agreeing rollout $y^+$ is promoted to stand in for the gold solution $y^\star$, and a subset $\mathcal{B}^-_x \subseteq \mathcal{Y}^-_x$ of the disagreeing rollouts becomes the distillation target.

It is long, but it reads as one sentence: take the rollouts that fell outside the majority one at a time, show both sides the first $n$ tokens of one — but let the teacher $\bar\pi$ also see the majority answer $y^+$ — then pull the two next-token distributions together under a divergence $D_\beta$, averaged over every token and every disagreeing rollout. $\mathcal{U}$ is the pool of unlabeled problems, $y^-_{<n}$ is a prefix of a disagreeing rollout, $|y^-|$ is that rollout's length in tokens, $\theta$ are the trained parameters, and $\bar\pi$ is the same network with gradients detached. Strip the notation away and what is left is a single instruction, one which says: at every step the model took down a wrong path, compare what it thought at that moment with what it would have thought had it known the group's answer — and close the gap. The two expectations stacked at the front mean only "average this over all the problems, and over the rollouts you drew for each one," and the two fractions in the middle mean only "count each disagreeing rollout once, and each of its tokens once," so that a long answer does not outweigh a short one. Two kinds of prompt are skipped: those where $c(x) < \tau$, so the vote cannot be trusted, and those where $\mathcal{Y}^-_x$ is empty, so there is nothing to correct.