#moe
8 articles
01
·Inference & Serving·★ MEMBER·PAPER·15 min read
Paper Walkthrough — SMELT: Is Looping the Same Layers Twice Actually a Win When the Budget Is Matched?
A study of looped Transformers that finally controls the comparison: per-token FLOPs, total parameters, and KV cache are all held close. The resulting recipe, SMELT, loops the middle half twice and reports 6.8–18.0% training-FLOPs savings on the compute-optimal frontier.
02
·How Transformers Work·★ MEMBER·PAPER·16 min read
Paper Walkthrough: Designing Qwen3.8-Next — Accuracy, Efficiency and Stability as One Problem
A ground-up read of the Qwen3.8-Flash-Next design report: the GDN hybrid, Qwen Sparse Attention, the Gated Residual and n-gram embeddings, judged the way the paper judges them — on loss, on cost, and on stability.
03
·Model Families·★ MEMBER·9 min read
The Mistral Family from Scratch — Europe's Small-and-Strong Bet
How a Paris startup made 'small but strong' work: the sliding-window attention behind Mistral 7B, the sparse mixture of experts behind Mixtral, the Codestral code line, and the licence policy that started at Apache 2.0 and then wandered — explained from no prior knowledge.
04
·Agents·★ MEMBER·PAPER·13 min read
Paper Explained: FreeToken — Treating Your Own PC as a Single Elastic Inference Platform
FreeToken is an edge-native serving system for frontier-scale MoE models on personal hardware. Its centerpiece is a q* policy that decides how many missed experts to ship over PCIe versus execute in place on the CPU — using nothing but two measured bandwidths.
05
·Model Families·FREE·9 min read
The DeepSeek Family from Scratch — Breaking In with MoE and Distillation
DeepSeek redrew the open-LLM map with four tools: MoE, MLA, GRPO, and distillation. Walk the V2/V3/R1 lineage from zero prerequisites, learn how to read that famous training-cost number, and find out what the distilled models are actually for.
06
·★ MEMBER·PAPER·10 min read
Paper Deep-Dive: The 'Physics' of Multimodal Pretraining — Which Way Does Knowledge Actually Flow?
What happens when a single model learns text and images together from day one? A large-scale controlled study from Meta FAIR and collaborators maps the asymmetric flow of knowledge between modalities, the conditions for synergy, the 'vision laziness' caused by late integration, and a recipe that gets strong image generation from just 5% of the data budget.
07
·Model Families·★ MEMBER·7 min read
The Qwen Family from Scratch — Why It Tops Hugging Face's Download Charts
Alibaba Cloud's Qwen is a fixture at the top of Hugging Face's download charts. From Qwen2.5's every-size-plus-specialist catalog strategy to Qwen3's thinking mode and MoE, this guide covers the lineage, why it excels at multilingual and code work, and the pitfalls that bite in production — assuming zero prior knowledge.
08
·Paper Deep-Dives·★ MEMBER·PAPER·9 min read
Mixture of Experts (MoE) from Scratch — Routing and Load Balancing in the Switch Transformer
A ground-up explanation of Mixture of Experts, the sparse architecture behind today's largest LLMs, built strictly from the Switch Transformer paper: the router math, expert capacity, the load-balancing loss, and the three tricks that make sparse training stable.