Paper Walkthrough: On-Policy Self-Distillation in Diffusion Models — Turning Reward into a Target You Can Aim At

When you train an image generator with rewards, a score on the finished picture never tells the model how to change its intermediate denoising predictions. DiffusionOPSD builds explicit targets from reward gradients and fits them using an EMA copy of the model itself. A ground-up explanation, strictly within what the abstract states.

On-Policy Self-Distillation in Diffusion Models

"Make it better" doesn't fix a drawing

Picture a drawing class. You spend an hour on a piece, the instructor glances at it, says "62 out of 100," and walks away. You know it wasn't good. But you have no idea which line to move, in which direction, or by how much. Your next attempt scores 62 again.

That is exactly the situation when you train a diffusion model with reinforcement learning. RL can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change (abstract). The paper we're reading today — "On-Policy Self-Distillation in Diffusion Models" (arXiv:2608.24646, submitted 25 Aug 2026) — proposes DiffusionOPSD, a framework that turns the instructor who only says "62" into one who points at the canvas and says "this part, this way."

One disclaimer up front. The primary source available for this article covers the abstract only — the fetched full text did not include the paper's body sections. So everything I attribute to the paper below comes from the abstract; anything else is flagged as my own reading or practitioner advice.

Background: a diffusion model repaints a "finished guess" at every step

Diffusion generation isn't one shot. It starts from pure noise and removes a little of it at a time, over anywhere from a handful to dozens of steps. At each of those steps, the model internally holds a guess at the finished image: "if I stripped all remaining noise right now, this is what I'd get." That is what the paper calls the clean-output prediction.

In the notation common to diffusion models, given the noisy image $x_t$ at time $t$, the clean guess $\hat{x}_0$ is recovered as follows. (This is standard diffusion background, not a formula proposed by this paper.)

$$\hat{x}_0 = \frac{x_t - \sqrt{1-\bar{\alpha}_t}\,\hat{\epsilon}_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}$$

Symbol by symbol: $x_t$ is the half-finished, noisy picture; $\hat{\epsilon}_\theta$ is the noise the model believes is sitting on top of it; and $\bar{\alpha}_t$ is a coefficient describing how much noise is mixed in (closer to 1 means cleaner). Put plainly: subtract the noise you estimated, then divide to undo the fading, and out comes the finished image the model is currently imagining. Every sampling step is a small revision to that imagined picture. For the wider picture of how diffusion works, see Diffusion Models: an Introduction.

What's actually wrong with endpoint rewards

Here's the mismatch. That finished guess exists at every step, but the reward model — an aesthetic scorer, an instruction-following scorer — grades only the single image that comes out at the end. One scalar has to be spread thinly across dozens of predictions.

That hurts in two ways. Credit assignment: if the score is 62, was it the composition decided at step 3 or the texture at step 30? Nothing in the signal says. And variance: the same model scores differently on different seeds, so reading "should I change course?" out of score fluctuations takes a lot of samples.

This is the paper's starting point. Can an evaluation be translated into an instruction for intermediate predictions? DiffusionOPSD uses reward gradients to construct a concrete image that the current prediction should aim toward, and hands that over as supervision (abstract). Note the shift in the question being asked: not "was the result good?" but "what should this particular prediction have been?"

But gradients come with a trap. A gradient tells you the direction and nothing about how far to go. Push too hard and things break. In the figure below, the direction toward the valley is always right, yet as you raise the step size the ball leaps clean over the valley and diverges. When the paper says the positive and negative targets are constructed as bounded, this is the failure mode being controlled.