Students Professional
JA EN

#on-policy-distillation

5 articles

01 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of Steps Trained on a single query, on-policy distillation still improves for hundreds of steps and recovers most of full-data OPD's gain. The paper explains this with state coverage and absorption rate, and concludes OPD is data-overfed but algorithm-starved. 02 ·Agents·★ MEMBER·PAPER·10 min read Paper Walkthrough: SecOPD — Grading One Token at a Time to Cut Adaptive Prompt Injection by an Order of Magnitude Defensively fine-tuned LLMs still fall to adaptive prompt injections at close to 100%. The culprit is training that hands the whole response a single score. SecOPD grades every token using a teacher that never saw the injection — explained from first principles. 03 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement A third to a half of the teacher's token-level grades are wrong, yet the student improves just as fast either way. This paper traces on-policy distillation's gains not to imitating a teacher but to suppressing the student's own low-probability tokens — and drops the teacher entirely. 04 ·Training & Alignment·★ MEMBER·PAPER·10 min read Paper Walkthrough: PaperGym — Turning One Paper Into a Graded Training Environment for Research Plans A research plan has no answer key, so reinforcement learning has no environment. This paper splits each paper into four drawers and builds the question and the grading criteria from different drawers, producing 20,000 training environments where paraphrasing the prompt no longer earns reward. 05 ·Inference & Serving·★ MEMBER·PAPER·13 min read Paper explained: Self-OPD — an image generator that distills itself, with no teacher No teacher model at all: at every denoising step the student spawns K copies of itself, scores them, and pulls toward the good branches while pushing away from the bad ones. A ground-up walkthrough of Self-OPD, an alignment method for flow matching image generators.