Paper Explained: DataFlex-RL — Did 11 RLVR Data Policies Beat Uniform Sampling? Re-measured Over 12 Seeds

"Train only on the good rollouts and the model gets stronger" is fast becoming RLVR folk wisdom. DataFlex-RL (arXiv:2609.06107) compares 13 selection, reweighting and mixture-adaptation configurations under one GRPO recipe and 12 seeds, and reports that every difference from uniform sampling has a 95% interval that crosses zero. We walk through the paper, including the finding that the choice of evaluation benchmarks reshuffles the winners.

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

A paper that doubts "choose cleverly and you get stronger"

The paper is "DataFlex-RL: An Evaluation Platform for RLVR Data Policies" (arXiv:2609.06107; Liang, Chen, Feng, Qiang, Zhang — Peking University and others).

Here is the abstract in plain terms. In reinforcement learning with verifiable rewards (RLVR), a "data policy" decides which rollouts get used for training, how strongly each is weighted, and which domain the next batch of problems is drawn from. The authors built DataFlex-RL, a platform for comparing these policies under a single GRPO recipe, and ran 13 configurations × 12 seeds × 12 benchmarks (maths, logic, science) on Qwen2.5-7B-Base. Uniform GRPO raised average accuracy over the untrained model by 7.76 points, but for all eight selection and reweighting methods the paired 95% confidence interval against uniform sampling contained zero, and none of the three adaptive mixtures beat a fixed equal mixture. Extending to Llama-3.1-8B-Base with 12 seeds still produced no consistent winner. On top of that, re-scoring with a "maths-heavy six" set that drops the logic benchmarks gives a method ranking that is negatively correlated with the 12-benchmark ranking (ρ = −0.33). The conclusion: changing the data policy does change the training process, but it did not produce a reproducible improvement over uniform training.

This is not a flashy new-method paper. It belongs to a different genre: "we measured the tricks everyone believes in, under matched conditions, and the gap disappeared." The more you run RLVR yourself, the more a result like this is worth reading.

An analogy: how a tutor picks problems

Picture a cram school that improves students (the model) by having them work through a problem book. The tutor has three kinds of levers.

1. Choose which worked answers get marked (drop the answer sheets that are all correct or all wrong, since marking them teaches little)
2. Change how much weight each marked answer gets (spend twice as long on the near misses, say)
3. Change which subject's problems come next (if maths is stalling, give more maths)

All three feel intuitively right. The paper's question is simple: if you compare these three levers with the same problem book, the same tutor and the same marking scheme, do they really beat doing nothing special — marking every answer equally and splitting subjects evenly? (§1)

Background: where "data policies" appear in RLVR and GRPO

Let's start from zero. RLVR (Reinforcement Learning with Verifiable Rewards) is a framework for training LLMs with reinforcement learning on problems whose answers can be checked mechanically — the final answer to a maths problem, the solution to a logic puzzle. No separate reward model has to be trained; a verifier returns a 0/1 reward. How this differs from RLHF, which learns from human preferences, is covered in Instruction tuning and RLHF from scratch.

GRPO (Group Relative Policy Optimization) is the best-known update rule for it, and the paper describes it in four steps (§1): sample prompts → generate several responses per prompt → score them with the verifier → compute advantage relative to the other responses to the same prompt. Because of this "compare within the group" design, a prompt whose five responses are all correct, or all wrong, yields no ranking and becomes a degenerate group that contributes nothing to the update. That gives rise to the idea "wouldn't it be more efficient to use only problems near the boundary?", and a crowd of methods followed: DAPO's accuracy filter, PODS's variance maximisation, Advantage Reweighting, DUMP's adaptive curriculum (§1).

The paper organises them by separating what is changed (the intervention point) from what the change is based on (the signal) (§3). There are three intervention points:

• Selection: choose which of the already-generated responses enter the update
• Reweighting: use every response, but continuously vary its contribution to the loss
• Mixture adaptation: change the proportion of domains from which the next rollout's prompts are drawn

Mechanism: three interventions in one loss

The reference step samples maths, logic and science prompts in equal shares, generates $K=5$ responses per prompt, verifies them, and applies the standard GRPO loss to every valid token (§3.1). Below, $g$ is a prompt, $k$ a response to that prompt, $\ell$ a token within the response, $L_{gk}$ the response length, and $\ell^{\mathrm{GRPO}}_{gk\ell}$ the standard per-token GRPO loss.

In words, selection uses a 0/1 mask on each response to decide whether it enters the update at all. A masked-out response disappears entirely, and one with mask 1 keeps its usual GRPO loss.

Selection simply uses a per-response 0/1 mask $m_{gk}$ to decide "this answer goes to marking / doesn't". A mask of 0 removes the response entirely; a mask of 1 leaves the usual GRPO loss as is. The baseline sets every mask to 1. The paper's selection methods are difffilter, maxvar, gfpo and topk.

In words, reweighting keeps every response but multiplies each token's loss by a non-negative weight, so some tokens count more and others less. With every weight equal to 1, this reduces to the standard GRPO loss.

Reweighting replaces the mask with a non-negative weight $w_{gk\ell}$ multiplied onto each token. Some methods use one weight per response, others vary it per token. The baseline sets every weight to 1, and the paper's methods normalise the weights to average 1. The methods are ar, per, softmax and diffband.

In words, the loss is the ordinary GRPO loss for each domain, averaged using the domain probabilities p_t(d). Only those proportions change from step to step, which determines how many problems come from each domain.

Mixture adaptation changes the probability $p_t(d)$ of picking domain $d$ at each step $t$. What sits inside the expectation is the ordinary GRPO loss; the only thing that moves is the proportion of problems drawn from each domain. With three domains, the fixed control is $(1/3,1/3,1/3)$. The methods are reward_gap, dump_ucb and tscl.

Every mixture method has the same shape: turn a per-domain score $u_d$ into probabilities with a softmax. dump_ucb, for instance, uses $p_t(d)\propto\exp(u_d/T)$ with temperature $T=1$ and a floor of 0.05 (Table 1). How the temperature sharpens that split clicks once you play with the figure below.

One important caveat (end of §3.1). Selection and reweighting act after the rollouts, so they do not reduce generation cost. Moreover, selection is not normalised, so the number of tokens used in the update itself shrinks, whereas reweighting keeps the average at 1. In other words, when selection "seems to work", you cannot tell whether that is the effect of targeting or merely the effect of a different amount of training. The paper separates the two later with control experiments.