self-distillation

Definition

Distillation where the teacher is the model itself — an earlier checkpoint, an ensemble of its own predictions, or its outputs filtered by a verifier — rather than a separate larger model. The appeal is improving without new human labels or a bigger teacher, which is why it recurs in self-training pipelines. The obvious risk is reinforcing the model's own errors, so filtering quality does most of the work.

Explainers using this term

• Paper Explained: Negative Self-Distillation — Learning to Reason by Steering Away From Your Worst SelfNegative Self-Distillation: Learning to Reason by Avoiding Flaws
• Paper Walkthrough: DART-SD — Training Tool-Calling Agents Without Flattening the DiamondDART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents
• On-Policy Distillation — Learning From What the Student Actually WritesOn-Policy Self-Distillation without Any Supervision