#distillation

36 articles

01 ·Time Series·★ MEMBER·PAPER·7 min read Paper Explained: Video Generation Models: A Survey of Post-Training and Alignment — Four Toolboxes for Making Video Models Do What You Ask A from-scratch reading of the survey that organizes how pretrained video generators are steered toward instruction-following, temporal coherence, physical plausibility, and safety without retraining—via supervised fine-tuning, self-training and distillation, preference and reward optimization, and inference-time methods. 02 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Explained: Beyond Teacher Assignment — Why Choosing the Right Teacher Isn't Enough When You Distill Many Specialists into One Model When MOPD distills math, code and instruction-following specialists into one student, the instruction-following teacher shouts so loudly that math barely transfers. DN-MOPD measures each teacher's feedback spread and evens out the volume, with a one-line change. 03 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Explained: Post-Training Leaves Behavioral Shadows on Unrelated Decisions — capability transfer through one unrelated word Ask a code-tuned model to pick between 'jacket' and 'tie' a few thousand times, and another model can inherit part of its coding gain. A from-scratch walkthrough of Active Taskless Distillation (ATD): the math, the controls, and the limits. 04 ·How Transformers Work·★ MEMBER·PAPER·9 min read Paper Deep Dive: Your Transformer Can Hold Two Thoughts at Once — Blend Two Texts and the LLM Still Remembers Both Average the embeddings of two unrelated texts, run a single forward pass, and an LLM's output still keeps both streams' next-token candidates near the top. We walk through the paper's evidence that this 'superposition linearity' fades during pretraining, comes back with light fine-tuning, and can be partially decoded into two continuations. 05 ·Inference & Serving·★ MEMBER·PAPER·7 min read Paper Explained: When EOS Tokens Disagree — Why On-Policy Distillation Students Forget How to Stop In on-policy distillation, student responses can grow until they hit the generation budget. This paper (arXiv:2609.20511) traces a major cause to student and teacher marking 'the end' with different special tokens. We walk through it from scratch, using only the paper itself. 06 ·Inference & Serving·★ MEMBER·PAPER·8 min read Paper Explained: Negative Self-Distillation — Learning to Reason by Steering Away From Your Worst Self Self-distillation from ground-truth solutions (OPSD) can strip a reasoning model of its habit of pausing and double-checking. NSD has the model play a 'careless reasoner' and trains only to move away from that bad teacher — no labels required. A from-scratch walkthrough. 07 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: X-AuT — Pull Audio-Encoder Layers Out Two at a Time, Then Backfill from a Bigger Teacher Delete whole layers from a speech LLM's audio encoder and the decoder starts guessing 'end of sentence' too early, dropping characters. X-AuT picks which layers to remove with short trial runs, then backfills in three stages: distillation from a larger teacher, supervision on the student's own decoding context, and LoRA finetuning. Qwen3-ASR-0.6B goes from 18 to 16 layers with macro-average error falling 5.61% → 5.27%, and 14 layers land at 5.75% — single-run numbers, read here through the equations and the recipe. 08 ·★ MEMBER·PAPER·12 min read Paper Deep Dive: Mask Forcing — Fixing Autoregressive Video Diffusion Distillation with a Dual-Noise Mask Distilling a bidirectional video diffusion teacher into a real-time autoregressive student tends to produce over-saturated, over-smoothed video. Mask Forcing eases this mode collapse by randomly swapping some tokens in the student's own rollout for slightly cleaner ones — no real data, no extra post-training stage. 09 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Walkthrough: On-Policy Reverse Distillation — Borrowing a Weak Teacher's Direction, Not Its Answers OPRD takes the policy change a weak teacher acquired during post-training and uses it only as a direction to amplify the student's own policy gradient — never as an optimization target. A ground-up walkthrough. 10 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Walkthrough: AuK — Folding Speech Generation and Editing Into One Model AuK is an open-source foundation model that puts speech generation, editing, enhancement and separation behind one door: a natural-language instruction plus optional audio. A ground-up walkthrough of its data design, MMDiT backbone, rectified flow, human-preference and GRPO post-training, and four-step distillation. 11 ·Inference & Serving·★ MEMBER·PAPER·11 min read Paper Walkthrough: One Training Example Keeps On-Policy Distillation Improving for Hundreds of Steps Trained on a single query, on-policy distillation still improves for hundreds of steps and recovers most of full-data OPD's gain. The paper explains this with state coverage and absorption rate, and concludes OPD is data-overfed but algorithm-starved. 12 ·★ MEMBER·PAPER·9 min read Paper Walkthrough: LLaDA-Image — Building the Visual Prior from Images Alone, Then Distilling to 2–4 Steps A training recipe that builds the visual prior from images alone before any caption enters the picture. We walk through LLaDA-Image — a 6B DiT trained from scratch and distilled down to 2–4 sampling steps — following the design decisions the paper actually makes. 13 ·Inference & Serving·★ MEMBER·PAPER·10 min read Paper Explained: Compile by Training — Turning a Natural-Language Spec into a Function That Runs Locally "Sort my email into urgent and later" — a spec that vague, turned into a small function you can run locally after about a minute of training. A ground-up walkthrough of Compile by Training: analogy, mechanism, equations, measured numbers. 14 ·★ MEMBER·PAPER·14 min read SolarWM: Training on 5 Seconds, Walking for an Hour — and Opening the Whole Stack A data engine that folds 10 datasets into 1.43M clips under one contract, plus a three-stage recipe that adapts four different video backbones without erasing them. The causal models roll out for minutes to hours after training on 5-second sequences alone. 15 ·Agents·★ MEMBER·PAPER·13 min read Paper Walkthrough: UI-Venus-2 — Taking Screen-Operating Agents From Benchmarks to Real Work A ground-up read of the UI-Venus-2 Technical Report: a GUI agent that drives phones, browsers and desktops from pixels alone. What it scales, how it verifies its own training data, and where it still falls short. 16 ·Agents·★ MEMBER·PAPER·15 min read Paper Walkthrough: Training Agents to Evolve with Their Harness Why compact models break when you rewrite prompts and tool definitions every week, and what the team behind Taobao Live's AI avatar streamers does about it. A ground-up walkthrough of Harness-Aware Training, from intuition to equations to measured numbers and stated limits. 17 ·★ MEMBER·PAPER·14 min read Paper Walkthrough: DreamX-Creator — Making Sound and Picture Together in 7B, Then Finishing at 2K in One Step A ground-up walkthrough of a 7B model that denoises audio and video inside one generative process: the gated cross-modal attention, the modality-aware reinforcement learning, the one-step 2K refiner — and the unusually heavy caveats the authors put on their own results. 18 ·Inference & Serving·★ MEMBER·PAPER·12 min read Paper Walkthrough: DART-SD — Training Tool-Calling Agents Without Flattening the Diamond When a multi-turn tool-calling task has order-independent sub-goals, the set of correct solutions spreads out into a diamond lattice. This walkthrough explains why whole-trajectory imitation crushes that structure, and how locating the first point where a rollout leaves recoverable territory — and supervising only what comes after it — changes the picture. 19 ·Inference & Serving·★ MEMBER·PAPER·11 min read TTPO Explained: Training a Model Mid-Exam, With No Answer Key Majority-vote pseudo-labels are wrong on roughly 85% of competition math prompts. So why does training still work? A ground-up walkthrough of TTPO's asymmetric design: distill the rollouts that agree, penalize the ones that disagree. 20 ·Distillation & Compression·★ MEMBER·PAPER·9 min read On-Policy Distillation — Learning From What the Student Actually Writes Classic distillation has the student copy sentences the teacher wrote. On-policy distillation has the student write, then lets the teacher mark it up. The difference is one symbol in the loss — and that symbol removes exposure bias, turns distillation into a form of RL, and opens the door to self-distillation methods like u-OPSD and AgentOPSD. 21 ·Distillation & Compression·★ MEMBER·PAPER·10 min read Distillation vs. Quantization vs. Pruning — Three Roads to a Smaller Model There are three roads to a smaller model: coarsen the number grid (quantization), remove weights outright (pruning), or rebuild the thing at a smaller size (distillation). A head-to-head comparison on compression ratio, accuracy, and implementation cost — and why, when you stack them, training-based methods go first and quantization goes last. 22 ·Distillation & Compression·FREE·PAPER·9 min read A Field Guide to Distillation Recipes — logit, feature, attention, self Every distillation recipe is an answer to one question: which part of the teacher should the student match? This guide lines up output (logit), intermediate feature (FitNet), attention, and self-distillation in one table and four equations, then turns the choice into a decision you make from constraints — is the teacher behind an API, do the layers line up, do the head counts match. 23 ·Distillation & Compression·FREE·PAPER·9 min read The Math of Distillation — Why Soft Answers Teach More Why is the distillation loss KL(teacher||student), what is the temperature T actually doing, and where does that mysterious T² factor in every implementation come from? A ground-up derivation of why a soft distribution carries more teaching signal than a correct answer. 24 ·Distillation & Compression·★ MEMBER·PAPER·11 min read Distilling Agents — How to Compress a Long Trajectory When you distill an agent, the unit of learning changes from a single question-and-answer to an entire game. Tens of thousands of tokens come back with one bit of grading. This article works through how that gap is bridged — turn-level credit assignment, trajectory filtering and on-policy correction, and the inheritance of tool use — and maps the recent long-horizon agent literature onto those three axes. 25 ·Distillation & Compression·★ MEMBER·PAPER·8 min read When Distillation Fails — Capacity Gaps and Contagious Overconfidence A stronger teacher does not guarantee a stronger student. A student too small to hold the teacher's structure returns a blurred average; a confident teacher hands down its errors along with its confidence; and the student walks paths the teacher never traveled. The three ways distillation breaks, from first principles. 26 ·Distillation & Compression·★ MEMBER·PAPER·9 min read Evaluating Distilled Models — Is "Close to the Teacher" a Good Metric? Score a distilled student by how often it agrees with its teacher and the students that faithfully reproduce the teacher's mistakes come out on top. What agreement actually guarantees, what breaks outside the training distribution, and how to test for contamination that arrives paraphrased through synthetic data. 27 ·Distillation & Compression·FREE·PAPER·8 min read Designing Distillation Data — Deciding What to Ask the Teacher What decides a distilled student's quality is less how smart the teacher is than what you asked the teacher to answer. Synthetic data generation, coverage design, why you should skew toward hard problems, correctness filtering, and why DeepSeek-R1's distillation worked — from first principles. 28 ·Distillation & Compression·★ MEMBER·PAPER·11 min read Build Your Own Distillation — Growing a Small Model in 100 Lines The distillation loss fits in twenty lines — and almost everyone who writes it trips on the same three things: the direction of the KL, the choice of reduction, and the missing T². We build the whole rig: freezing the teacher, the loss, the training loop, the teacher-free baseline, a temperature sweep, and four sanity checks that prove the implementation isn't quietly broken. 29 ·RAG & Retrieval·★ MEMBER·PAPER·13 min read Paper Walkthrough: WeMM-Embedding — Putting Text, Images and Video on One Ruler A walkthrough of WeMM-Embedding (2B/4B/9B), Tencent's universal multimodal embedding family. The unified pair format, the <embedding> token, Matryoshka dimensions, two-stage training and distillation — explained from scratch, using only the numbers the paper reports. 30 ·Large Language Models·FREE·PAPER·10 min read Knowledge Distillation from Scratch — Copying a Big Model into a Small One No label ever says "dogs look a bit like cats." A trained model's output does — and that gap is what knowledge distillation harvests. Soft labels, temperature-scaled softmax, where the T² factor comes from, sequence-level and synthetic-data distillation, and how DeepSeek-R1 copied an entire reasoning procedure into smaller models. 31 ·Inference & Serving·FREE·11 min read Cutting Inference Cost in Practice — What to Do First Quantization and distillation both work, but in most shops there are two or three levers left that cost you nothing in quality. Break the bill into four numbers, then reorder the work: cache, batch, shorten, compress. What each lever actually buys, and what breaks when you take them out of order. 32 ·Model Families·FREE·9 min read The DeepSeek Family from Scratch — Breaking In with MoE and Distillation DeepSeek redrew the open-LLM map with four tools: MoE, MLA, GRPO, and distillation. Walk the V2/V3/R1 lineage from zero prerequisites, learn how to read that famous training-cost number, and find out what the distilled models are actually for. 33 ·FREE·PAPER·12 min read Paper Review: S²VOPD — Don't Strengthen the Teacher, Blur the Student's View A walkthrough of S²VOPD, which improves vision-language models without labels, rewards, or a stronger teacher. The trick is inverting where the asymmetry comes from: degrade the student's view instead of privileging the teacher's. 34 ·Inference & Serving·FREE·PAPER·14 min read Paper Walkthrough: No Gold Answers, No Stronger Teacher — How u-OPSD Distills From Its Own Majority Vote u-OPSD (arXiv:2608.06296) replaces the gold solution in on-policy self-distillation with the model's own majority vote, then corrects only the rollouts that disagree with it. A from-scratch walkthrough grounded solely in the paper. 35 ·FREE·PAPER·8 min read Paper Explained: JoyAI-Video-Edit — Autoregressive Diffusion That Keeps Editing 720p Video at ~30 FPS, Without Ever Seeing the Future Say 'swap the background for snowy mountains' and the live feed comes back edited as it streams — a walkthrough of JD's JoyAI-Video-Edit from the paper itself. We trace the three-stage recipe (chunk-wise autoregression, Source-Anchored DMD, long-horizon distillation) that lets a 16B model edit 720p video at roughly 30 FPS. 36 ·Inference & Serving·FREE·PAPER·8 min read Paper Walkthrough: DAPD — Breaking the Teacher's "Cheat-Sheet Illusion" in Distillation with Dual Anchors When you distill from a teacher that can see the reference solution, the student learns to act as if invisible answers were still there — the privilege illusion. DAPD (arXiv:2608.01735) traces this failure to information asymmetry and fixes it with two levels of anchoring. A walkthrough grounded strictly in the paper itself.