born-again

Definition

Self-distillation in which the student has the same architecture and size as the teacher — no compression, just retraining on the teacher's soft outputs. The surprising empirical result is that such students often exceed the teachers they came from, usually explained by the teacher's output distribution being a softer, more informative target than one-hot labels. The procedure can be iterated across generations.