on-policy

Definition

A reinforcement learning regime where the data used for an update comes from the current policy itself. Because the training distribution matches the policy being improved, updates are better behaved, but every update invalidates the collected data, so sample efficiency is poor. Off-policy methods reuse old or externally generated data via replay buffers or importance weighting, trading stability for efficiency.

Explainers using this term

• Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of StepsRethinking On-Policy Distillation of Large Language Models II: One Training Example
• Paper Walkthrough: SecOPD — Grading One Token at a Time to Cut Adaptive Prompt Injection by an Order of MagnitudeSecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation
• Paper Explained: Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-ImprovementDoes On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement