#soft-label

1 articles

01 ·Large Language Models·FREE·PAPER·10 min read Knowledge Distillation from Scratch — Copying a Big Model into a Small One No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.