Dense vs mixture-of-experts: what you trade, from routing to serving
The architecture-level comparison: how a router picks experts, why load balancing is the hard part, capacity factors and dropped tokens, how much of each model actually runs per token (from Mixtral's 28% to DeepSeek-V3's 5.5%), what MoE buys in training compute, and what it costs in memory, communication and agent behaviour.
- deep dive
- MoE
- architecture
- training
A dense model uses every parameter for every token. A mixture-of-experts (MoE) model routes each token through a small subset. It’s the most important architectural trade in current LLMs, and the right answer depends on whether you’re paying for training, memory or bandwidth. The single-GPU MoE note covers serving on one card; this one covers the architecture itself.
The router
In an MoE layer, the feed-forward block is replaced by E experts, each a feed-forward network of its own, plus a router. For each token, the router computes a score per expert, keeps the top k, and combines the chosen experts’ outputs weighted by their scores:
scores = softmax( W_router · x ) one score per expert
y = Σ over top-k experts score_e · Expert_e(x)Attention layers usually stay dense. The idea is old: Shazeer et al. (2017) scaled noisy top-k gating to thousands of experts. GShard (Lepikhin et al., 2020) and the Switch Transformer (Fedus et al., 2021, which used k = 1) made it work at transformer scale.
How much runs per token
| Model | Total parameters | Active per token | Active fraction |
|---|---|---|---|
| Mixtral 8x7B (8 experts, top-2) | 46.7B | 12.9B | ~28% |
| Qwen3.6-35B-A3B | ~35B | ~3B | ~9% |
| DeepSeek-V3 (256 routed + 1 shared, top-8) | 671B | 37B | ~5.5% |
The trend is clear: more, smaller experts and a smaller active fraction. Compute per token follows active parameters. Memory follows total parameters.
The hard part: load balancing
Left alone, routers collapse. A few experts get good early, get chosen more, get trained more, and get better still, while the rest starve. Two standard countermeasures:
- An auxiliary loss. The Switch Transformer adds
α · E · Σ f_e · P_e, wheref_eis the fraction of tokens sent to experteandP_eits mean router probability. It’s minimised when load is uniform. It works, but it competes with the language-modelling objective. - Capacity limits. Each expert gets a fixed buffer per batch, about
(tokens / experts) × capacity factor. Tokens that overflow an expert’s capacity are dropped from that layer and pass through only the residual connection. A higher capacity factor drops fewer tokens but wastes compute on padding.
DeepSeek-V3 avoided the auxiliary loss with a routing bias per expert, adjusted after each step to push load toward balance without touching the training loss (the DeepSeek note has the details). Fine-grained experts and an always-on shared expert also help: common knowledge lives in the shared expert, and many small experts can specialise.
What MoE buys
Training compute per unit of quality. At a fixed compute budget, a sparse model can hold far more parameters than a dense model and reach a lower loss. The Switch Transformer reported reaching the same quality several times faster than a dense model with the same compute per token. That’s why the largest open models are MoE: their capacity is set by memory, but their training and inference cost per token is set by the active parameters.
What MoE costs
- Memory. All experts must be stored, so a 671B MoE needs the memory of a 671B model, even though it computes like a 37B one.
- Communication. In distributed training and serving, experts live on different GPUs, and every MoE layer needs an all-to-all exchange to send tokens to their experts and back. Much of DeepSeek-V3’s engineering went into hiding that traffic.
- Batching efficiency. Different tokens choose different experts, so a large batch touches many experts and loses part of the batching advantage dense models get.
- Training instability. Router collapse, loss spikes and sensitivity to precision are common. Recipes are more fragile than for dense models.
- Behaviour. On my own agent benchmark, Qwen3.6-35B-A3B scored 742.2 with a loop-discipline score of 65.9, against 816.2 and 89.0 for the dense Qwen3.6-27B. That’s one harness and one model pair, but it’s a reminder to measure on your task (the leaderboard has all ten models).
Choosing
| Situation | Lean towards |
|---|---|
| Training budget is the constraint, memory is plentiful | MoE |
| Serving many users across a cluster | MoE, with expert parallelism |
| One GPU with little VRAM, plenty of system RAM | MoE with expert offload |
| One GPU, enough VRAM, agentic tool use | Dense: test both, as my benchmark suggests |
| Small fine-tuning projects | Dense: simpler to train and to adapt |
Rules I use
- Quote both numbers: total parameters for memory, active for compute and speed.
- Watch expert load when fine-tuning an MoE. Imbalance is the first sign a recipe is breaking.
- Benchmark behaviour, not just loss. Equal perplexity doesn’t mean equal tool discipline.
Sources: Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017). Lepikhin et al., GShard (2020). Fedus, Zoph and Shazeer, Switch Transformers (2021). Jiang et al., Mixtral of Experts (2024). Dai et al., DeepSeekMoE (2024). DeepSeek-AI, DeepSeek-V3 Technical Report (2024). The benchmark numbers are from my amber-bench leaderboard.