#moe

11 articles

01 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: IntBMoE — Every Expert Contributes, Yet Each Token Runs Only a Few Blocks In a Mixture-of-Experts layer, how many experts inform a token, how many get computed, and how many weight sets must be built have always been tied together. IntBMoE (arXiv:2609.21346) cuts those ties by composing experts densely and executing blocks sparsely. Explained from scratch, using only the paper itself. 02 ·Inference & Serving·★ MEMBER·PAPER·6 min read Paper Explained: DeepSeek-V4.1-Flash — How Far Can KV Cache Compression Go? DeepSeek-V4.1-Flash shrinks its global KV cache to 890 bytes per token. We unpack the four ideas behind it (the CED layer split, cross-layer KV sharing in CSA2, FP4 caching, and SWA Bounded Replay) from first principles. 03 ·★ MEMBER·PAPER·17 min read Paper Walkthrough: Miles v0.1 — Anatomy of a Post-Training Stack Built to Survive Production A read-through of Miles, a full-stack system for agentic RL post-training of frontier MoE models: how it hunts down the numerical drift between the rollout engine and the trainer, and what a 744B model actually measured on 64 GPUs. 04 ·Inference & Serving·★ MEMBER·PAPER·16 min read SMELT explained (arXiv:2609.01343) — compute-matched looping Does running the same layers twice help once the budget is matched? SMELT holds per-token FLOPs, total parameters and KV cache within about 4%, loops the middle half twice, and reports 6.8-18.0% training-FLOPs savings on the compute-optimal frontier. 05 ·How Transformers Work·★ MEMBER·PAPER·16 min read Qwen3.8-Next design report explained (arXiv:2608.30320) A ground-up read of the Qwen3.8-Flash-Next report: 125B total parameters with 6B activated, the GDN hybrid, Qwen Sparse Attention (7.6x faster prefill at 1M context), the Gated Residual and n-gram embeddings — judged on loss, cost and training stability. 06 ·Model Families·★ MEMBER·9 min read The Mistral Family from Scratch — Europe's Small-and-Strong Bet How a Paris startup made 'small but strong' work: the sliding-window attention behind Mistral 7B, the sparse mixture of experts behind Mixtral, the Codestral code line, and the licence policy that started at Apache 2.0 and then wandered — explained from no prior knowledge. 07 ·Agents·★ MEMBER·PAPER·13 min read Paper Explained: FreeToken — Treating Your Own PC as a Single Elastic Inference Platform FreeToken is an edge-native serving system for frontier-scale MoE models on personal hardware. Its centerpiece is a q* policy that decides how many missed experts to ship over PCIe versus execute in place on the CPU — using nothing but two measured bandwidths. 08 ·Model Families·FREE·9 min read The DeepSeek Family from Scratch — Breaking In with MoE and Distillation DeepSeek redrew the open-LLM map with four tools: MoE, MLA, GRPO, and distillation. Walk the V2/V3/R1 lineage from zero prerequisites, learn how to read that famous training-cost number, and find out what the distilled models are actually for. 09 ·FREE·PAPER·10 min read Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow? What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget. 10 ·Model Families·★ MEMBER·7 min read The Qwen Family from Scratch — Why It Tops Hugging Face's Download Charts Alibaba Cloud's Qwen is a fixture at the top of Hugging Face's download charts. From Qwen2.5's every-size-plus-specialist catalog strategy to Qwen3's thinking mode and MoE, this guide covers the lineage, why it excels at multilingual and code work, and the pitfalls that bite in production — assuming zero prior knowledge. 11 ·Paper Deep-Dives·FREE·PAPER·9 min read Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable.