tinybert

Definition

A compact Transformer produced by distilling BERT, from Huawei's Noah's Ark Lab. Its distinguishing choice is two-stage distillation — once during general pretraining and again on the downstream task — with losses that match embeddings, attention matrices, and hidden states rather than only the output logits. It is a standard reference point for BERT compression, alongside DistilBERT and MiniLM.