Paper Explained: Bellman Policy Optimization — Policy Mirror Descent for LLMs, Without a Value Model

In LLM reinforcement learning where only the final answer is scored, BPO (arXiv:2609.15987) targets the same optimum as Policy Mirror Descent without estimating a single intermediate state value. We walk from the Bellman 'telescoping sum' to the practical loss, using only the paper itself as the source.

Bellman Policy Optimization

An exam graded only on the final answer

Picture a math teacher who never reads your working and only checks whether the final answer is right. You write hundreds of lines, and what comes back is a single bit: right or wrong. Nobody tells you which lines were good and which ones sank you.

That is exactly the setting of RLVR (Reinforcement Learning with Verifiable Rewards), the recipe used to push LLM reasoning with reinforcement learning. On tasks like math, where an answer can be checked mechanically, the reward lands only at the very end of the response.

The paper "Bellman Policy Optimization" (Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing; arXiv:2609.15987, 2026-09-14) turns that "reward only at the end" property into an advantage.

Here is the abstract in plain terms. RLVR improves LLM reasoning. The authors propose BPO, a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, the Bellman equations let them rewrite PMD as a trajectory-level objective (one term per whole response), so no state values need to be estimated at intermediate states. They prove this rewrite has the same unique optimal solution as the original PMD objective, then approximate it to obtain a practical BPO loss, whose correction weight is a smoothed ratio of complementary token probabilities (1 − p). Experiments on math reasoning benchmarks show it works.

Refresher: what GRPO does

The workhorse of today's RLVR is GRPO (Group-Relative Policy Optimization) (§2.3). For each problem it samples G responses, standardizes their rewards within the group, and calls the result the advantage $\hat{A}^i$ — how much better than average that response was.

In words: solve the same problem 16 times, give a plus to the attempts that did better than the group average and a minus to the ones that did worse. Every token in a response shares that one score.

There is a practical wrinkle. The policy that generated the responses (the rollout policy $\mu$) and the policy being trained ($\pi$) drift apart slightly, because of update lag and numerical differences between the inference and training engines (§2.3). GRPO corrects for this with the importance ratio $r=\pi(y_t)/\mu(y_t)$ and clips it PPO-style. That $r$ is precisely the piece BPO ends up replacing.

PMD: "improve the good moves, but don't wander far"

BPO starts from Policy Mirror Descent (PMD) (§2.5). At every state it solves:

In words: raise the probability of good moves (tokens with large advantage $A^\mu$), but pay a penalty — the KL divergence — for moving too far from the old policy $\mu$. The step size $\eta$ sets how bold each move is. If KL divergence is new to you, see our KL divergence article.

This problem has a closed-form answer (Eq. 17):

In words: take the old probability, multiply by the exponential of "how good," and renormalize so everything sums to one. $Z_\mu(s)$ is just that normalizing total (the partition function). This is a softmax, with $\eta$ playing the role of an inverse temperature. In the widget below, lowering the temperature — the same as raising $\eta$ — makes the distribution pile onto the best options.

The catch is that you need $A^\mu(s,y)$ at every intermediate state. To know how good each single token was, at every position of a response hundreds or thousands of tokens long, you either train a separate value model (more memory, more compute) or re-roll from that point many times and average (expensive). The paper also cites prior work showing learned value estimates can be inaccurate on reasoning tasks (§1).

The key observation: advantages telescope