Paper Explained: IntBMoE — Every Expert Contributes, Yet Each Token Runs Only a Few Blocks
In a Mixture-of-Experts layer, how many experts inform a token, how many get computed, and how many weight sets must be built have always been tied together. IntBMoE (arXiv:2609.21346) cuts those ties by composing experts densely and executing blocks sparsely. Explained from scratch, using only the paper itself.
Key points
- IntBMoE is an MoE design that decouples the three quantities of participation, execution and materialization for a single token.
- Participation is how many experts contribute knowledge, execution is how many are computed (compute cost), and materialization is how many expert-sized parameter sets are built and stored (memory cost).
- IntBMoE pairs dense expert composition with sparse block execution and beats representative sparse and dense MoE baselines on image classification.
- Sparse routing sends a token only to the k highest-scoring experts; unpicked experts don't affect the output and get no learning signal.
- It is fully deployed in AMap's recommender, where it delivered a 2.4% relative UVCTR gain within a 60 ms latency budget.
IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
Primary source — what this article is built on
undefined2026-09-18→undefined2026-09-25same month
IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-ExpertsRan Cheng, Longfei Xu, Zheng Liu et al. · 2026-09-18 · v1arXiv:2609.21346Paper page·PDFundefined
Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.
A kitchen analogy
Picture a kitchen with 16 cooks. Call all of them for every order and each plate carries everyone's skill, but labor costs 16x. Call only two and it's cheap, but fourteen talented people sit idle.
Now suppose that before opening, all 16 cooks work out 8 house recipes together, and each order is filled by cooking just two of them. Every recipe carries all 16 cooks' know-how, each order costs only two dishes of work, and the fridge never holds more than 8 prepared bases. That is the core idea of "IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts" (arXiv:2609.21346, DreamX at Alibaba Group, posted 2026-09-18).
The abstract, in short: for a single token, an MoE has three quantities: participation (how many experts contribute knowledge), execution (how many are actually computed, which sets compute cost), and materialization (how many expert-sized parameter sets must be built and stored, which sets memory cost). Existing designs can't set these independently. IntBMoE pairs dense expert composition with sparse block execution to decouple all three. It beats representative sparse and dense MoE baselines on image classification and also works for language modeling and sequential recommendation. It is fully deployed in AMap's recommender, where it delivered a 2.4% relative UVCTR gain within a 60 ms latency budget.
If MoE itself is new to you, start with our Mixture of Experts explainer.
Three existing approaches, three bottlenecks
The first is sparse routing (§3):
In words: send the token only to the highest-scoring experts, then add their outputs weighted by the routing probabilities . Experts that aren't picked don't affect the token's output and get no learning signal from it (§1).
The second, dense output mixing, computes every expert, so compute grows with the number of experts. The third, parameter merging (SMEAR and others), averages all experts' weights into one expert and runs it once. But each routing unit needs its own merged copy, so materialization balloons when merging per token (§3).
The widget below shows how a router's softmax turns scores into probabilities. Sparse routing keeps only the top .
Comments
Sign in to comment