#mixture-of-experts

4 articles

01 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: IntBMoE — Every Expert Contributes, Yet Each Token Runs Only a Few Blocks In a Mixture-of-Experts layer, how many experts inform a token, how many get computed, and how many weight sets must be built have always been tied together. IntBMoE (arXiv:2609.21346) cuts those ties by composing experts densely and executing blocks sparsely. Explained from scratch, using only the paper itself. 02 ·Inference & Serving·★ MEMBER·PAPER·16 min read SMELT explained (arXiv:2609.01343) — compute-matched looping Does running the same layers twice help once the budget is matched? SMELT holds per-token FLOPs, total parameters and KV cache within about 4%, loops the middle half twice, and reports 6.8-18.0% training-FLOPs savings on the compute-optimal frontier. 03 ·Audio & Speech·★ MEMBER·PAPER·10 min read Paper Walkthrough: SwanTale — Designing Voices from Words Alone, with Speech and Sound in One Waveform ByteDance's SwanTale handles two jobs in one model: cloning a voice from reference audio (zero-shot) and building voices, ambience, and sound effects purely from a natural-language caption (instruct). We walk through the paper, from the 70M-caption data pipeline to Unified MoE routing and GRPO post-training. 04 ·Paper Deep-Dives·FREE·PAPER·9 min read Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable.