self-distillation

Definition

Distillation where the teacher is the model itself — an earlier checkpoint, an ensemble of its own predictions, or its outputs filtered by a verifier — rather than a separate larger model. The appeal is improving without new human labels or a bigger teacher, which is why it recurs in self-training pipelines. The obvious risk is reinforcing the model's own errors, so filtering quality does most of the work.

Explainers using this term

• Paper Walkthrough: DART-SD — Training Tool-Calling Agents Without Flattening the DiamondDART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents
• Paper Walkthrough: On-Policy Self-Distillation in Diffusion Models — Turning Reward into a Target You Can Aim AtOn-Policy Self-Distillation in Diffusion Models
• On-Policy Distillation — Learning From What the Student Actually WritesOn-Policy Self-Distillation without Any Supervision