#pruning

2 articles

01 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: X-AuT — Pull Audio-Encoder Layers Out Two at a Time, Then Backfill from a Bigger Teacher Delete whole layers from a speech LLM's audio encoder and the decoder starts guessing 'end of sentence' too early, dropping characters. X-AuT picks which layers to remove with short trial runs, then backfills in three stages: distillation from a larger teacher, supervision on the student's own decoding context, and LoRA finetuning. Qwen3-ASR-0.6B goes from 18 to 16 layers with macro-average error falling 5.61% → 5.27%, and 14 layers land at 5.75% — single-run numbers, read here through the equations and the recipe. 02 ·Distillation & Compression·★ MEMBER·PAPER·10 min read Distillation vs. Quantization vs. Pruning — Three Roads to a Smaller Model There are three roads to a smaller model: coarsen the number grid (quantization), remove weights outright (pruning), or rebuild the thing at a smaller size (distillation). A head-to-head comparison on compression ratio, accuracy, and implementation cost — and why, when you stack them, training-based methods go first and quantization goes last.