TTPO Explained: Training a Model Mid-Exam, With No Answer Key

Majority-vote pseudo-labels are wrong on roughly 85% of competition math prompts. So why does training still work? A ground-up walkthrough of TTPO's asymmetric design: distill the rollouts that agree, penalize the ones that disagree.

TTPO: Test-Time Policy Optimization

Getting better mid-exam, with no answer key

Picture yourself halfway through a mock exam. There is time left on the clock, but nobody will hand you the marking scheme. You can still do something useful: solve the same problem several different ways, treat whichever answer came up most often as "probably the right one," and use it to audit your own reasoning.

Machine learning calls this test-time training (TTT) — improving on the very problems you are being asked to solve, without ever seeing the labels. It throws away the usual assumption that training data and deployment data are separate things, and uses the test set itself as the training set.

TTPO (Test-Time Policy Optimization) brings that setting to mathematical reasoning in large language models. The claim is blunt: without using a single ground-truth label, it matches or beats methods that do use them (§1).

The setup: both mainstream post-training recipes demand labels

The first family is reinforcement learning from verifiable rewards (RLVR). Let the model solve the problem, give it a reward of 1 if the final answer matches the ground truth and 0 otherwise. But that reward is a single scalar per rollout, broadcast uniformly across every token, so it never says which particular step was the good one. The paper calls it "coarse" (§1).

The second is on-policy self-distillation (OPSD), and the idea is stranger. You build a teacher by conditioning the same model on the answer. The teacher, knowing where the solution lands, re-reads the student's own text token by token and produces a distribution saying "here is what I would have written." Where RL gave one number per rollout, this gives a supervision signal at every token — far denser (§2.2).

The problem is that both need the answer: RLVR to verify, OPSD as privileged context for the teacher. In TTT, where labels never arrive, neither transfers (§1).

What breaks when the majority vote becomes the teacher

If there is no label, have the model make one. Sample several rollouts per problem and take the most common answer as a pseudo-label — a majority vote. TTRL used exactly this as a reward and improved reasoning with no labels at all (§2.1).

So why not drop the pseudo-label into OPSD wherever the ground truth used to go? Here the paper lands a sharp line: a corrupted reward misleads once per trajectory; a corrupted teacher misleads at every token (§1). Dense supervision is powerful when it is right, and equally dense in its damage when it is wrong.

A note before we go on: a teacher's "distribution" is peaked when the model is confident and flat when it is unsure. That peakedness becomes the heart of TTPO later, so get a feel for it with your hands first.

The paper's observation: the errors are asymmetric

Then come the measurements. Running test-time training on Qwen3-1.7B over AIME 2026, the pseudo-label is wrong for roughly 85% of prompts (§3.2, Figure 1a). A teacher that is wrong four times out of five sounds unusable.

But the same measurement yields a second fact. Even when the pseudo-label is wrong, about 79% of the rollouts that disagreed with it produce an answer that is neither the pseudo-label nor the ground truth (§3.2).

That is the crux. Penalizing a disagreeing rollout never touches the content of the pseudo-label. It uses one bit only: this rollout is not in the majority cluster. So the penalty is usually correct whether or not the vote was. Distilling toward the pseudo-label has no such tolerance — it consumes the answer itself, so the error flows straight into the teacher distribution.

Penalties do not depend on the pseudo-label; distillation does. That asymmetry is TTPO's entire design principle. The paper ties it to negative learning from noisy labels, where saying "this is not a dog" stays reliable long after "this is a cat" has stopped being (§1).

Put differently: one signal, the majority vote, is decomposed into a trustworthy use and an untrustworthy one. Prior methods applied that signal uniformly to every rollout, so the blast radius of a bad vote scaled with the number of rollouts. TTPO confines the branch where errors can enter to the agreeing rollouts only — and, as the next section shows, arranges for the error to be defanged even there. What follows is the actual loss, and then the token-level selection layered on top of it.

The mechanism: distill the agreers, penalize the disagreers

For a problem $x$, sample $K$ rollouts and extract their final answers $a_k$. Cluster answers by mathematical equivalence, take the largest cluster as the pseudo-label $\hat{a}$ (§3.1), then split the rollouts in two.

$\mathcal{P}$ is the set of rollouts that agreed with the vote; $\mathcal{N}$ the set that disagreed.

Put in words: the $K$ rollouts are sorted into two piles on a single bit — did this one land on the majority answer, or not. Nothing about the reasoning inside a rollout, or where it went off the rails, enters the split.