#on-policy-distillation

8 articles

01 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Explained: Beyond Teacher Assignment — Why Choosing the Right Teacher Isn't Enough When You Distill Many Specialists into One Model When MOPD distills math, code and instruction-following specialists into one student, the instruction-following teacher shouts so loudly that math barely transfers. DN-MOPD measures each teacher's feedback spread and evens out the volume, with a one-line change. 02 ·★ MEMBER·PAPER·8 min read Paper Explained: SenseNova-U1.5 — One Model That Both Sees and Draws, With No Encoder and No VAE SenseNova-U1.5 is an 8B unified model that learns straight from pixels and words, with no vision encoder for understanding and no VAE for generation. We walk through its seam-free spatial decoder and its 'train specialists separately, then distill them into one' post-training, following the paper itself. 03 ·Large Language Models·★ MEMBER·PAPER·13 min read Paper Explained: FlowBalance — Verifier-Grounded Self-Improvement from a Model's Own Reasoning Learning from your own answers breaks in two ways: the grading is too sparse, or you believe your own confidence. FlowBalance fixes both with one move — let the verifier decide the sign of the dense self-score — and learns a normalized distribution over whole responses. Explained from first principles. 04 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of Steps Trained on a single query, on-policy distillation still improves for hundreds of steps and recovers most of full-data OPD's gain. The paper explains this with state coverage and absorption rate, and concludes OPD is data-overfed but algorithm-starved. 05 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: SecOPD — Grading One Token at a Time to Cut Adaptive Prompt Injection by an Order of Magnitude Defensively fine-tuned LLMs still fall to adaptive prompt injections at close to 100%. The culprit is training that hands the whole response a single score. SecOPD grades every token using a teacher that never saw the injection — explained from first principles. 06 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement A third to a half of the teacher's token-level grades are wrong, yet the student improves just as fast either way. This paper traces on-policy distillation's gains not to imitating a teacher but to suppressing the student's own low-probability tokens — and drops the teacher entirely. 07 ·Training & Alignment·★ MEMBER·PAPER·11 min read Paper Walkthrough: PaperGym — Turning One Paper Into a Graded Training Environment for Research Plans A research plan has no answer key, so reinforcement learning has no environment. This paper splits each paper into four drawers and builds the question and the grading criteria from different drawers, producing 20,000 training environments where paraphrasing the prompt no longer earns reward. 08 ·Inference & Serving·★ MEMBER·PAPER·13 min read Paper explained: Self-OPD — an image generator that distills itself, with no teacher No teacher model at all: at every denoising step the student spawns K copies of itself, scores them, and pulls toward the good branches while pushing away from the bad ones. A ground-up walkthrough of Self-OPD, an alignment method for flow matching image generators.