minilm

Definition

A small language model from Microsoft Research built by deep self-attention distillation: the student is trained to reproduce the teacher's last-layer attention distributions and the relations among its value vectors. Because supervision is confined to a single layer, the student need not mirror the teacher's depth or width. The distillation is task-agnostic, so the result serves as a general-purpose compact backbone.