#mixture-of-experts
3 articles
01
·Inference & Serving·★ MEMBER·PAPER·15 min read
Paper Walkthrough — SMELT: Is Looping the Same Layers Twice Actually a Win When the Budget Is Matched?
A study of looped Transformers that finally controls the comparison: per-token FLOPs, total parameters, and KV cache are all held close. The resulting recipe, SMELT, loops the middle half twice and reports 6.8–18.0% training-FLOPs savings on the compute-optimal frontier.
02
·Audio & Speech·★ MEMBER·PAPER·10 min read
Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform
ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training.
03
·Paper Deep-Dives·★ MEMBER·PAPER·9 min read
Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer
A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable.