2026-08-31·★ MEMBER·PAPER·13 min read

Paper explained: Self-OPD — an image generator that distills itself, with no teacher

No teacher model at all: at every denoising step the student spawns K copies of itself, scores them, and pulls toward the good branches while pushing away from the bad ones. A ground-up walkthrough of Self-OPD, an alignment method for flow matching image generators.

#flow-matching #on-policy-distillation #reward-model #image-generation #rl

ModalitytextTaskinference

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Primary source — what this article is built on

undefined2026-08-27→undefined2026-08-31same month

Self-OPD: On-Policy Distillation for Flow Matching Models without TeacherShiyi Zhang, Mushui Liu, Yunze Tong et al. · 2026-08-27 · v1arXiv:2608.26872Paper page·PDF

undefined

On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce \textbf{Self-OPD}, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into $K$ stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.

The phrase that shouldn't make sense: distillation without a teacher

Say "distillation" and most people picture a clever teacher model whose outputs a smaller student learns to imitate. The teacher is what makes the supervision dense: at every single step, the student is told what it should have done. That density is the whole appeal of on-policy distillation (OPD).

Carry that idea into image generation, though, and it gets expensive. Rendering text correctly, respecting the composition you asked for, looking good to a human — each new objective means training another specialised teacher. And the student can never beat its teacher: the teacher's quality and its biases become the ceiling (§1).

Self-OPD asks the obvious follow-up question. Can we keep the dense per-step supervision of OPD and delete the teacher?

The picture: send scouts down every fork in the trail

Imagine walking down a mountain. The destination is "a good image."

Teacher-based OPD is a guide walking beside you, pointing at every fork. Convenient, but you pay for the guide, and you can never go anywhere the guide doesn't know. Reinforcement learning is the opposite: you walk all the way down and only then hear "that route scored 70." Which fork actually hurt you is anybody's guess, so the gradients are noisy (§1).

Self-OPD sits between the two. At every fork it spawns K copies of itself, nudges each one in a slightly different direction, and runs all of them to the bottom to be scored. It also records what "just walking straight ahead" would have scored, and subtracts that as a baseline. Branches that beat the baseline attract; branches that lose to it repel. No guide required — only a scorer.

Background: flow matching learns a velocity field

Flow matching (FM) models learn the velocity of a flow that carries noise into an image (§3.1). Here $t=1$ is pure noise and $t=0$ is the image. During training you build an intermediate state $x_t=(1-t)x_0+t\epsilon$ and ask the model which way it should be moving.

\mathcal{L}_{\mathrm{FM}}=\mathbb{E}_{t,x_{0},\epsilon}\left[\|v_{\theta}(x_{t},t)-(\epsilon-x_{0})\|^{2}\right]

(1)

$v_\theta(x_t,t)$ is the velocity the model predicts, and $(\epsilon-x_0)$ is the true direction from noise toward data. Written out in words, equation (1) says: show the model a half-noisy picture, ask it to point at the way back toward a real image, and penalise it by how far the arrow missed. The model never memorises pictures; it just plays that arrow game a few million times. At generation time you integrate that velocity from $t=1$ down to $t\approx0$ (an ODE). Add a little noise at each step and you get the SDE version, which makes the same model produce a different image every time.

x_{t_{j+1}}=x_{t_{j+1},\theta}+\sigma_{t_{j}}\sqrt{|\Delta t_{j}|}\,z_{j},\quad z_{j}\sim\mathcal{N}(0,\mathbf{I})

(2)

$x_{t_{j+1},\theta}$ is the deterministic next state the model computed, $z_j$ is Gaussian noise, and $\sigma_{t_j}=\eta\sqrt{t_j/(1-t_j)}$ sets how strong that noise is. With $\eta=0$ you fall back to the plain ODE. Put in words, equation (2) says: don't take the point the model picked; roll a die and land somewhere around it instead. How far around is set by $\eta$ — and that radius will shortly become the range of our scouting party.

Where the two existing approaches got stuck

The paper sorts prior work into two camps (§1, Fig. 1a). RL methods such as Flow-GRPO need no teacher, but the reward only arrives at the end of the trajectory, so credit has to be pushed back through dozens of steps and the gradients come out noisy. Teacher-based OPD (Flow-OPD, DiffusionOPD) regresses onto a teacher velocity at every step, which is stable, but every objective needs its own teacher, and blending several teachers' velocity fields at the field level makes the updates fight each other. DiffusionOPD in particular routes each prompt to the teacher that matches it — which, the paper argues, is at odds with the actual goal of making one image satisfy every criterion at once.

The shared idea: tilting a distribution with reward

One concept runs through all three approaches. The goal is to move the model's current next-step distribution $q_\theta$ toward a version tilted slightly in favour of high reward, $q^{*}\propto q_{\theta}\exp(A/\tau)$ (§3.3). Here $A$ is how good a candidate was and $\tau$ is a temperature controlling how aggressively you lean toward the winner. Large $\tau$ leaves the candidates nearly level; small $\tau$ bets everything on the single best one.

FIG 1Lower the temperature and probability piles onto one branch. Self-OPD's reward-tilted target has the same shape — too low a temperature means betting everything on the best branch, which destabilises training

Mechanism 1: branching, and scoring against yourself

A single Self-OPD step is three operations (§3.2).

(a) One forward pass, K branches. Compute the deterministic next state $x_{t_{j+1},\theta}$ exactly once, then add K independent Gaussian perturbations to it. The expensive transformer evaluation happens only once; all the diversity comes from cheap noise. That is the efficiency trick.

x_{t_{j+1}}^{(k)}=x_{t_{j+1},\theta}+\sigma_{t_{j}}\sqrt{|\Delta t_{j}|}\,z_{k},\quad z_{k}\sim\mathcal{N}(0,I)

(3)

Equation (3) has the same shape as (2). In plain terms it says that the branches $k=1,\dots,K$ all scatter from one shared point, differing only in which noise draw $z_k$ they got — K possible futures for the price of a single expensive forward pass.

(b) Roll each branch out and score it. Every branch is completed to a clean latent $\hat{x}_0^{(k)}$ by a deterministic ODE rollout ( $\eta{=}0$ ), decoded by the VAE, and scored by the task reward model.

(c) Use yourself as the baseline. From the same parent state, also run straight ahead without branching and record that score, $r^{\mathrm{ode}}$ . This is the self-reference baseline — no teacher, no second model.