Paper Walkthrough: No Gold Answers, No Stronger Teacher — How u-OPSD Distills From Its Own Majority Vote

u-OPSD (arXiv:2608.06296) replaces the gold solution in on-policy self-distillation with the model's own majority vote, then corrects only the rollouts that disagree with it. A from-scratch walkthrough grounded solely in the paper.

On-Policy Self-Distillation without Any Supervision

A study room where someone took the answer key away

Picture two students in a study room who happen to be the same person. One of them — the teacher — has the worked solution in hand and solves the problem while looking at it. The other — the student — sees only the problem statement. Now force the student to copy, token by token, what the teacher believes the next move should be. That is on-policy self-distillation (OPSD). Teacher and student share weights, which is why it is called "self" distillation. But what makes the teacher smarter is the worked solution, and that came from outside.

The paper pushes on exactly that point: does the teacher in on-policy self-distillation actually need a ground-truth solution, or can a model build its own privileged context and perform genuine self-distillation (§1)? The answer is u-OPSD. In place of the gold solution it drops in the majority vote over eight of the model's own attempts at the same problem. All it consumes is the problem statement — no gold labels, no environmental feedback, no larger teacher model.

The map: what each recipe was borrowing from outside

GRPO's weakness is that its signal is sparse: a rollout thousands of tokens long receives one bit of information. OPD and OPSD made it dense, telling the model at every single token what the distribution should have been. But as the paper points out, OPSD is self-distillation only in the sense that the parameters are shared — the information that makes the teacher capable still arrives from outside (§1). That is what capped these methods at labeled problems.

The core idea: one sample is unreliable, agreement among eight is a signal

The observation is simple: "although an individual rollout may be unreliable, agreement among multiple independently sampled rollouts provides an endogenous confidence signal" (§1). Wrong answers scatter. A correct chain of reasoning tends to converge on one value, while mistakes land wherever the model happened to slip, so if six of eight rollouts land on the same answer, that answer is worth more than any single one of them.

There is a second payoff. The rollouts that fall outside the majority are, by construction, a list of the places worth fixing. Distillation is applied only there. In the paper's words, this lets the model "correct itself precisely where it is confidently wrong" (Abstract).

Three steps: sample, vote, distill

Algorithm 1 lays out the per-prompt procedure in three stages (§3.2).

(1) Sample. For each problem $x$, draw $G$ rollouts at the training temperature from $\bar\pi$, a stop-gradient copy of the current policy. Pull the final answer $a^{(g)}$ out of \boxed{...} with normalization and canonicalization. Anything unparsable — usually a truncation — is marked invalid.

(2) Vote. The plurality among the valid answers becomes the pseudo-answer $\tilde a(x)$, with ties broken uniformly at random. That splits the rollouts into an agreeing set $\mathcal{Y}^+_x$ and a disagreeing set $\mathcal{Y}^-_x$. Invalid rollouts join neither, because a generation that was cut off is evidence of neither a correct nor an incorrect belief. Confidence in the vote is measured by the self-consistency score:

$c(x)$ is the self-consistency score, $G$ the number of rollouts, $a^{(g)}$ the answer the $g$-th rollout ended up with, $\tilde a(x)$ the answer that won the vote, and $\mathbb{1}[\cdot]$ a counter that returns 1 when the condition inside holds and 0 otherwise. Put in words, it says: count how many of the eight attempts came back with the same answer as the winner, then divide by eight. Equation (1) is a show of hands written as a fraction, and nothing more. One detail matters more than it looks: the denominator is $G$, the total drawn, not the number of valid ones. A prompt that truncates often loses confidence for that reason alone and gets filtered out automatically (§3.2).

(3) Distill. Only prompts whose $c(x)$ reaches the threshold $\tau$ are used. One agreeing rollout $y^+$ is promoted to stand in for the gold solution $y^\star$, and a subset $\mathcal{B}^-_x \subseteq \mathcal{Y}^-_x$ of the disagreeing rollouts becomes the distillation target.

It is long, but it reads as one sentence: take the rollouts that fell outside the majority one at a time, show both sides the first $n$ tokens of one — but let the teacher $\bar\pi$ also see the majority answer $y^+$ — then pull the two next-token distributions together under a divergence $D_\beta$, averaged over every token and every disagreeing rollout. $\mathcal{U}$ is the pool of unlabeled problems, $y^-_{<n}$ is a prefix of a disagreeing rollout, $|y^-|$ is that rollout's length in tokens, $\theta$ are the trained parameters, and $\bar\pi$ is the same network with gradients detached. Strip the notation away and what is left is a single instruction, one which says: at every step the model took down a wrong path, compare what it thought at that moment with what it would have thought had it known the group's answer — and close the gap. The two expectations stacked at the front mean only "average this over all the problems, and over the rollouts you drew for each one," and the two fractions in the middle mean only "count each disagreeing rollout once, and each of its tokens once," so that a long answer does not outweigh a short one. Two kinds of prompt are skipped: those where $c(x) < \tau$, so the vote cannot be trusted, and those where $\mathcal{Y}^-_x$ is empty, so there is nothing to correct.

As pseudocode, the whole per-prompt procedure is this short:

ys = [sample(pi_frozen, x) for _ in range(G)]      # (1) draw G rollouts
ans = [extract_boxed(y) for y in ys]               #     None means invalid
a_hat = plurality([a for a in ans if a is not None])
c = sum(a == a_hat for a in ans) / G               # (2) denominator is G, not len(valid)
if c < tau: return                                 #     vote not trusted
pos = [y for y, a in zip(ys, ans) if a == a_hat]
neg = [y for y, a in zip(ys, ans) if a not in (a_hat, None)]
if not neg: return                                 #     nothing to correct
y_plus = max(pos, key=len)                         # (3) the teacher's reference
loss = mean(fwd_kl(teacher(x, y_plus, y[:n]), student(x, y[:n]))
            for y in neg for n in range(len(y)))

This is also where u-OPSD separates itself from ordinary majority-vote self-training. The paper is explicit: it "does not simply imitate the selected response under teacher-forced prefixes. Instead, it transfers the solution-conditioned next-token distribution along the model's own disagreeing trajectories" (§3.2). Rather than copying out the winning answer, the model copies what a version of itself that already knows that answer would think while walking down the wrong path.

A teacher shown only the final answer is worse than the base model

The ablation that best exposes the design philosophy belongs here rather than later (§4.3, Figure 5). When the teacher's reference is stripped down to just the \boxed{} pseudo-label, performance drops by 10.3% to 15.8%, and all three label-only settings land below the base model's 43.57. The paper's conclusion: effective distillation requires the teacher to condition on the longer, full reasoning trace rather than only the final answer. What gets transferred is not the answer but the shape of thinking of someone who knows how the answer was reached.

Which rollout you pick matters much less. Choosing the longest, the shortest, or a random one puts seven of the nine full-reference settings within 2.1% of the best, and the best configuration — teacher on the longest agreeing rollout, distilling the longest disagreeing one — scores 59.00.

What the experiments actually measured

Qwen3-4B and 8B in both non-thinking and thinking modes, plus Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. The same checkpoint serves as both student and teacher. Training data is a 30k subset of OpenThoughts, and u-OPSD uses only the problem statements, never the solution field. The baselines are trained on identical prompts: three supervised ones that do use gold data (SFT, GRPO, OPSD) and three label-free self-rewarding RL methods (TTRL, RENT, Intuitor). Evaluation covers AIME24, AIME25 and HMMT25 at avg@12 and MATH500 and AMC23 at avg@4, decoded with vLLM at temperature 1.0 with a 38k generation cap (§4.1).

The training recipe largely follows OPSD: forward KL ($\beta=0$) over the full vocabulary with per-token pointwise clipping, a teacher frozen at the initial policy, LoRA of rank 64 ($\alpha=128$), learning rate $5\times10^{-6}$, sampling at temperature 1.1 with top-$p$ 0.95 and top-$k$ 20, and 150 steps with a checkpoint every 25. Only three settings are specific to u-OPSD: $G=8$, $\tau=0.5$, and raising the maximum completion length from 1,024 to 4,096 tokens.

Results: the label-free method beats the labeled ones

Non-thinking mode (Table 1) is where the gap is widest. On the five-benchmark average, Qwen3-4B goes from a base of 40.96 to 49.49 with u-OPSD, above SFT (42.73), GRPO (45.86) and OPSD (46.29); Qwen3-8B goes from 43.57 to 54.31, above OPSD's 52.04. The paper describes this as "8.5% and 10.7%" over the base and "3.2% and 2.3%" over OPSD (§4.2); as the raw table numbers show, these are differences in average accuracy — percentage points.

By contrast, the label-free RL baselines running on the same rollout budget gain at most 1.5% over the base models. The paper's reading is that consensus works better "as privileged teacher context for token-level distillation, rather than reducing them to scalar rewards" (§4.2). Same majority vote, different outcome depending on whether you compress it into one bit or hand it over as a distribution across the whole vocabulary.

Thinking mode (Table 2) narrows the gap: 74.85 → 77.05 at 4B and 76.09 → 77.99 at 8B, essentially tied with OPSD (76.20 / 77.97) and ahead of GRPO. The explanation offered is that consensus-based self-distillation works best when the base model is reliable enough to produce informative consensus while still leaving substantial headroom, and stronger thinking-mode models leave less. On the instruction-tuned models (Table 3), Qwen3-30B-A3B goes from 75.77 to 77.46 and Qwen3-4B-Instruct from 67.00 to 68.78 — the former offered as evidence that the same recipe transfers to a substantially larger mixture-of-experts model without model-specific tuning.

Which knobs actually matter

Including the places where the paper says outright that its own defaults are not optimal (all from §4.3, Qwen3-8B non-thinking):

• Threshold $\tau$: the ranking is monotone and looser is better. $\tau{=}0.3$ scores 58.59, the default 0.5 scores 57.10, and $\tau{=}0.9$ collapses to 44.40 — a 14.2% spread.
• Rollouts per prompt $G$: real returns, but they saturate fast. $G{=}4$ (56.99) and $G{=}8$ (57.10) are indistinguishable, $G{=}12$ gains 4.7% over the default, and $G{=}16$ gives half of that back. The default of 8 exists to keep generation cost matched with OPSD; the paper states plainly that $G{=}12$ is the better operating point when generation is not the constraint.
• Teacher update: the default freezes the teacher at the initial policy (under LoRA, the base weights with the adapter disabled). Letting it track the student via EMA does better — decay 0.995 gains 2.4% at the best checkpoint and 4.1% at step 150.
• Where the divergence is computed (Table 4): evaluating only at the token the student sampled scores 43.45, barely above the base model and not competitive. Full vocabulary reaches 57.10, and top-100 is the best at 59.01. Truncating to top-$k$ is a practical optimization that cuts cost without costing accuracy.
• Which divergence (Table 5): forward KL 57.10, symmetric JSD 43.34 (level with the untrained model), and reverse KL never converged.

The reverse-KL failure is documented concretely enough to be worth reading. Average generation length runs 2.7k characters at step 25, 12k at step 50, 76k at step 75, and 99k — the ceiling — from step 125 onward, while the fraction of rollouts with a parsable boxed answer falls from 99% to 33%. Inspecting the completions shows a loss of termination rather than of coherent reasoning: a phrase, a LaTeX command pair, or a nesting bracket repeated until the budget runs out.

How good are the pseudo-labels?

Measured on 64 training prompts under the full training configuration ($G{=}8$, a 4,096-token budget, $\tau{=}0.5$), 96.3% of rollouts yield a parsable boxed answer, 94.0% of prompts clear the self-consistency threshold and receive a pseudo-label, and 86.7% of those pseudo-labels match the gold answer from the data source (§4.3). Fewer than 10% of valid rollouts disagree with their prompt's vote, so the distillation set is small and concentrated. That means 13.3% of the supervision is simply wrong — and in non-thinking mode the method still beats OPSD trained on real gold solutions. That contrast is the most interesting thing in the paper.

What the paper does not establish

§5 is candid about the limits. Reported without embellishment:

• Scope. One model family (Qwen3, at 4B and 8B, in both reasoning modes) and one domain (competition mathematics) with automatically checkable final answers. The voting mechanism requires an extractable, canonicalizable answer; extending to open-ended generation would mean replacing exact-match voting with a softer notion of consensus.
• The advantage is regime-dependent. The margin over OPSD is 3.2% / 2.3% in Table 1 but 0.9% / 0.02% in Table 2. The authors explicitly ask readers to treat consensus as a strong replacement for gold solutions in the non-thinking regime they measured and as a match for them in the thinking regime, not as a uniform improvement.
• It is bounded by base-model competence. Majority-vote supervision reproduces whichever answer the base model already produces most often, so that answer's accuracy is the ceiling. The 13.3% wrong-pseudo-label figure was measured in-domain, but training dynamics under deliberately corrupted votes were not measured; detecting or down-weighting low-quality consensus by vote margin is named as a natural next step.
• Variance and repeats. Every checkpoint was evaluated with 12 samples across five benchmarks to control variance, but seed-replicated error bars for every training setup are still pending.

One reading of my own to add: the headline numbers are the best score across checkpoints evaluated every 25 steps, applied identically to u-OPSD and OPSD (§4.1). That is symmetric as a comparison, but in production you have to pick a checkpoint without a benchmark to select on, and the paper does not address that.

How this shows up in practice

Who, and when. A post-training engineer who has plenty of in-domain problem statements but no worked solutions and no verifier, and who cannot use a stronger model as a teacher for licensing reasons. All u-OPSD needs is problem statements plus an output format whose answer can be extracted mechanically. If gold solutions are plentiful, just run OPSD; this method earns its keep when labels are expensive, unreliable, or unavailable (§1).

Two numbers to measure before you commit. First, the base model's in-domain accuracy; second, its self-consistency rate when you draw eight rollouts per problem. The pattern in the paper — large gains in non-thinking mode, small ones in thinking mode — depends on sitting in the band where the model is smart enough to vote well and still fallible enough to have headroom. At either extreme (too weak and the vote is noise, too strong and there is nothing left to gain) the compute is not worth it.

Knobs to reach for first. (1) max_completion_length from 1,024 to 4,096 — a rollout that never reaches \boxed{} is invalid, and since $c(x)$ divides by $G$, that silently drags confidence down; tuning $\tau$ before fixing this is backwards. (2) $\tau$ at 0.3 rather than the default 0.5. (3) $G$ at 12 when generation is not the bottleneck — sampling cost is linear in $G$ and dominates the bill. (4) top-$k$ at 100 instead of the full vocabulary. (5) an EMA teacher at decay 0.995 instead of a frozen one.

Failure modes worth knowing about.

• Giving the teacher only the final answer is the worst thing you can do — it lands below the base model (a 10.3–15.8% drop). Pass the entire reasoning trace.
• Do not swap the objective. Reverse KL stops terminating and drives the parsable-answer rate from 99% to 33%; JSD lands at base-model level. Stay on forward KL with pointwise clipping.
• The effective batch is smaller than it looks. Under 10% of valid rollouts disagree with the vote, so drawing eight rollouts produces gradient on only a slice of them. Budget for "generation is expensive, training samples are few" up front.
• Prompts where the vote is confidently wrong push the model further in the wrong direction. Behavior under corrupted votes is unmeasured (§5), so if you ship this, plan on adding your own guard that drops prompts with a thin vote margin.

The question a design review will ask. "With no gold labels, why does this beat TTRL-style methods that turn the same majority vote into a reward?" The answer is what happens to the information: collapsed into a single scalar versus distributed across every token as a distribution over the vocabulary. In the paper's numbers, the former gains at most 1.5% over the base model and the latter gains 8.5–10.7% (§4.2).

Further reading

• DAPD — breaking the teacher's "cheat sheet illusion" — the same OPSD weakness attacked from the information asymmetry of the side that holds the answer
• AgentOPSD — recursive self-distillation for agentic RL — the line of work extending on-policy self-distillation to agents
• Instruction tuning and RLHF — the post-training map that GRPO and distillation sit inside