#deepseek-r1

2 articles

01 ·Distillation & Compression·FREE·PAPER·8 min read Designing Distillation Data — Deciding What to Ask the Teacher What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles. 02 ·Large Language Models·FREE·PAPER·10 min read Knowledge Distillation from Scratch — Copying a Big Model into a Small One No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models.