MoE on a single GPU: total vs active parameters, offload and bandwidth
A mixture-of-experts model costs memory by its total parameters and bandwidth by its active ones. What that means on one 24 GB card, how expert offload to system RAM changes the ceiling, why batching helps MoE less than dense models, and how MoE did in my agent benchmark.
- deep dive
- MoE
- inference
- llama.cpp
“35B-A3B” sounds like a contradiction until you split cost into two numbers. A mixture-of-experts model needs memory for all of its parameters, but each token only reads a few of them. On a single GPU, those two numbers pull in opposite directions.
Total vs active
In an MoE layer, the feed-forward block is replaced by many experts and a router. For each token, the router picks the top few experts and only those run. So:
- Total parameters decide memory: every expert must be loaded somewhere.
- Active parameters decide compute and, more importantly for decode, bytes read per token.
Take Qwen3.6-35B-A3B at roughly 4.8 bits per weight:
| Parameters | Bytes | |
|---|---|---|
| Must be stored | 35B | ~21 GB |
| Read per token | ~3B | ~1.8 GB |
A dense 27B reads 17 GB per token, so its bandwidth ceiling on an RTX 4090 is about 59 passes a second. The MoE reads about 1.8 GB, for a theoretical ceiling in the hundreds. In practice attention, routing and kernel overhead take over long before that. But it’s why MoE decode feels fast on modest hardware.
The memory problem, and offload
21 GB of weights on a 24 GB card leaves little room for the KV cache. The standard llama.cpp answer is to move expert weights to system RAM and keep everything else on the GPU: --cpu-moe for all experts, or --n-cpu-moe N for the first N layers. Attention and the shared layers stay on the GPU; the experts are read from RAM.
Now the token’s expert bytes come at system-memory speed. Dual-channel DDR5 delivers roughly 80 GB/s. If about 1.5 GB of expert weights per token are read from RAM, the ceiling is around 80 / 1.5 ≈ 53 tokens a second. That’s still respectable, and it’s why MoE models run surprisingly well on 8–16 GB cards with plenty of RAM. Partial offload, with only some layers’ experts in RAM, lands in between.
Batching helps less
For a dense model, batching is nearly free: every sequence uses the same weights, read once (the batching note has the numbers). In an MoE, different tokens route to different experts. As the batch grows, the set of experts touched in a pass grows too, and the bytes per pass climb back toward the total. MoE’s single-stream advantage partly disappears under load.
What my benchmark says
In amber-bench, Qwen3.6-35B-A3B scored 742.2, with a loop-discipline score of 65.9. The dense Qwen3.6-27B scored 816.2 and 89.0. The MoE also made 183 wasted tool calls, against 146 for the dense model. That’s one harness and one run, so it’s evidence, not a law. But it matches the pattern I see in agent work: fast tokens don’t help if the agent spends them going in circles.
When MoE is the right call
- Limited VRAM with plenty of system RAM: offload makes a large MoE usable where a dense model of similar quality wouldn’t fit.
- Single-user latency on modest hardware: few active bytes means fast decode.
- Agentic tool use: test it on your own harness before you commit. On mine, the dense 27B models were more disciplined than the MoE.
- High-concurrency serving: expert divergence erodes the batching advantage, so measure under realistic load.
Rules I use
- Size memory by total parameters and speed by active ones. Never the other way round.
- Offload experts before shrinking context. RAM bandwidth is often a better trade than a smaller KV cache.
- Benchmark the model in your own loop. Speed is not the metric for agents; completed tasks per hour is.
Sources: the offload flags are llama.cpp’s (--cpu-moe, --n-cpu-moe). The benchmark numbers are from my amber-bench leaderboard. The bandwidth ceilings are back-of-envelope: 1,008 GB/s for the RTX 4090, ~80 GB/s for dual-channel DDR5.