distilbert

Definition

A compact version of BERT released by Hugging Face, produced by knowledge distillation: a student with fewer transformer layers is pretrained to match the teacher BERT's output distribution rather than trained from scratch on labels alone. It retains most of the original's downstream quality while being smaller and faster to run, and it became one of the standard demonstrations that distillation transfers to large pretrained language models.