Paper Explained: Co-RL — Reasoning Without Labels, Emerging From a Diverse Cohort

Grade your own answers long enough and the model collapses. Co-RL breaks that loop by rewarding each agent against a peer's majority vote, matching supervised training without touching a single ground-truth label. The mechanism, the dynamics, the numbers, and the traps — straight from the paper.

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Why grading your own homework stops working

You solve a problem. You grade it yourself. The only reference is you. Do that a few hundred times — what happens?

Your score goes up. But the only thing going up is your score on the approaches you were already good at. The topics you misunderstood get reinforced, misunderstanding and all. A perspective you never had never shows up, no matter how many rounds you run.

That is exactly the shape of today's mainstream methods for reinforcement learning an LLM without labels. Co-RL changes one thing: the grader is not you, it is a classmate who grew up in a different school entirely. That single change, the paper argues, is enough to match label-supervised training while using zero ground-truth labels.

The labels run out before the compute does

The dominant recipe for improving reasoning is RLVR — reinforcement learning from verifiable rewards. Did the final answer match? Did the tests pass? A mechanical check produces the reward, and the reward updates the policy.

The catch is that the check is not free. The paper names two costs up front (§1). One is plain expense. The other is more fundamental: as reasoning capability approaches and passes what humans can reliably evaluate, ground-truth annotation itself becomes scarce. You cannot label a problem you cannot grade.

Hence self-rewarding RL: build the reward out of the model's own outputs instead of an external answer.

What self-rewarding RL actually does

TTRL, the canonical example, works like this (§3.2). Given an unlabeled prompt $x$, the policy $\pi_\theta$ samples $K$ completions. Each completion yields a final answer $a^k$, and the majority vote over those $K$ answers becomes the pseudo-label $\hat a_\theta(x)$. Agree with the vote, get reward 1; disagree, get 0.

$$r_{\mathrm{maj}}^{k}=\mathbf{1}\left[a^{k}=\hat{a}_{\theta}(x)\right]$$

Here $\mathbf{1}[\cdot]$ returns 1 when the bracketed statement is true and 0 otherwise — in words, twelve of your own answers said so, therefore whoever said so deserves credit. Intuitor rewards token-level confidence instead, and RENT rewards low predictive entropy, but they share the defining property — every bit of signal originates inside the same single model (§3.2).

With no external reference point, updates can only push up whatever answer is currently ahead. In the paper's framing: amplified biases and bad habits, shrinking response diversity, and eventually homogenized outputs and training collapse (§1).

Co-RL: borrow the reward from next door

Co-RL changes exactly one thing. The pseudo-label is built from another agent's rollouts, not your own (§4.1).

Take $N$ agents $\{\pi_{\theta_n}\}_{n=1}^N$ that share neither parameters nor gradients. Each independently samples $K$ completions for the same prompt $x$. The supervision target for agent $n$ is then

The formula says, in words, that agent $n$'s target is whichever answer its neighbor produced most often. The subscript $n-1$ carries the whole idea. Agent $n$ is taught by the majority vote of its neighbor $n-1$, never by its own. Indices wrap around, so agent 1 is supervised by agent $N$ — a directed ring. The reward is agreement with Eq. (1):

$$r_{n}^{k}=\mathbf{1}\left[a_{n}^{k}=\hat{a}_{-n}(x)\right]$$

No agent contributes to its own supervision target. That is the break with self-rewarding. From there each agent normalizes rewards within its own rollout group and updates only its own policy with GRPO; the overall objective is the average of the per-agent GRPO objectives (the paper's Eq. 4).

# one training step (pseudocode)
for x in batch:                                  # unlabeled prompts
    outs = [agent.rollout(x, K) for agent in agents]   # K each
    ans  = [[extract(y) for y in o] for o in outs]     # final answers
    pseudo = [majority(ans[n - 1]) for n in range(N)]  # neighbor's vote
    rew    = [[int(a == pseudo[n]) for a in ans[n]]    # 1 if it agrees
              for n in range(N)]
for n in range(N):                               # updates stay independent
    agents[n].grpo_step(outs[n], rew[n])

All rollouts and pseudo-labels are computed before any policy is updated (§4.1). Communication between agents happens only at the reward stage, and no external LLM judge or learned reward model is needed (§1).

Whether it helps depends on how differently they fail

A neighbor who is a copy of you is useless — you both miss the same problems the same way. So the paper pushes cohort diversity in three directions (§4.2).

1. Decoupled optimization. Two policies, separate parameters and optimizer state, no gradient crossing between them. Independent updates keep the two sets of predictions from becoming correlated.

2. Model family and size. This is the main lever. Qwen2.5 and Llama 3 differ in tokenizer, vocabulary size, architecture and pretraining corpus; Gemma brings a 256K-token vocabulary and interleaved local–global attention. For VLMs the gap is wider still, since the vision encoders themselves differ: a natively dynamic-resolution ViT in Qwen2.5-VL, InternViT in InternVL, SigLIP in Gemma 3.

3. Input formulation. Even with different models, both agents still read the same prompt string. So each MATH problem is rewritten by DeepSeek-V3, one agent trained on the original and the other on the rewrite. The rewrites are not lexical substitutions — they recast the problem into a different concrete scenario while preserving the answer, roughly doubling the question length (Appendix C).