#exposure-bias

1 articles

01 ·Distillation & Compression·★ MEMBER·PAPER·9 min read On-Policy Distillation — Learning From What the Student Actually Writes Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD.