Why it matters
Dense scaling raises compute and memory together, but quality tends to improve with parameters while serving cost tracks compute. A mixture of experts breaks that coupling: many more parameters are stored, yet each token is processed by only a couple of small networks. The price is a routing decision that must be learned, balanced and made deterministic enough for distributed training and batched inference.
Mental model
It is a hospital with many specialist departments and a triage desk: each patient sees two specialists, so the building is large but each visit is short. The metaphor breaks because triage is learned and differentiable only through the router's scores, and a popular department can be overwhelmed while others sit idle.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- top-k
- 2 of 5
- slots filled
- 8 × 2 = 16
- even share
- 16 ÷ 5 = 3.2
- expert 4 slots
- 0 of 16
Step 1 of 4
One row per token
The router scores every expert for every token before anything is dropped. Each row is a distribution over all five experts, so a low cell is still a considered score, not an absent one.
Next: Top two for t3
Figure. One mixture-of-experts layer with eight tokens and five experts, illustrative router probabilities. Each row is a token's softmax over every expert and totals 1.000; the top two cells in each row are the experts that run for that token. Column totals add the printed probabilities, so they measure routing mass rather than the number of tokens assigned.
What to notice. The top-two rule concentrates this batch: expert 2 takes 6 of the 16 slots, expert 1 takes 5, expert 5 takes 3, expert 3 takes 2 and expert 4 takes none, against 16 ÷ 5 = 3.2 slots if routing were even. Expert 4 also has the smallest column total, 0.690, so the imbalance is visible before any token is dropped at capacity. The probabilities are invented; the slot counts follow the rows exactly, and some implementations renormalise the two selected scores while others combine with the raw probabilities.
Text version
An eight by five matrix. Rows are tokens t1 to t8 and columns experts E1 to E5. Each cell prints a router probability and each row total reads 1.000. Row t1 prints 0.640, 0.100, 0.080, 0.060 and 0.120; t2 prints 0.580, 0.120, 0.090, 0.070 and 0.140; t3 prints 0.140, 0.520, 0.100, 0.080 and 0.160; t4 prints 0.160, 0.600, 0.110, 0.080 and 0.050; t5 prints 0.180, 0.550, 0.130, 0.090 and 0.050; t6 prints 0.120, 0.500, 0.220, 0.100 and 0.060; t7 prints 0.140, 0.480, 0.240, 0.090 and 0.050; t8 prints 0.180, 0.460, 0.160, 0.120 and 0.080. Column totals read 2.140, 3.330, 1.130, 0.690 and 0.710. Steps highlight the two selected cells for t3, the E2 and E4 columns, and the six E2 cells that show the load concentration.
- Replace the dense feed-forward block with a set of expert networks, each with its own parameters but the same shape.
- Compute a router score per token per expert, usually a linear projection followed by a softmax over experts.
- Select the top-k experts per token and run only those experts on that token.
- Combine the expert outputs weighted by their router probabilities, so the result stays differentiable with respect to routing.
- Apply a capacity factor and a load-balancing loss so no expert receives far more tokens than the average.
- All-reduce across devices when experts are sharded, since a token's chosen expert may live on another rank.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Top-1 routing | Cheapest per token, simple combine | Coarse specialisation, brittle to router noise | Tokens need more than one kind of transformation |
| Top-2 routing | Better quality, smoother gradients | Double the expert compute per token | Latency budget is tight |
| Expert parallelism | Models far larger than one device | All-to-all communication dominates | Interconnect is slow or batches are small |
| Large capacity factor | Fewer dropped tokens | More padding and wasted compute | Traffic is skewed across experts |
Good for
Cheapest per token, simple combine
Cost
Coarse specialisation, brittle to router noise
Fails when
Tokens need more than one kind of transformation
- Tokens need more than one kind of transformation
- Latency budget is tight
- Interconnect is slow or batches are small
- Traffic is skewed across experts
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Top-1 routing is good for Cheapest per token, simple combine, costs Coarse specialisation, brittle to router noise, and fails when Tokens need more than one kind of transformation. Top-2 routing is good for Better quality, smoother gradients, costs Double the expert compute per token, and fails when Latency budget is tight. Expert parallelism is good for Models far larger than one device, costs All-to-all communication dominates, and fails when Interconnect is slow or batches are small. Large capacity factor is good for Fewer dropped tokens, costs More padding and wasted compute, and fails when Traffic is skewed across experts
Failure modes
- Expert collapse. The router sends almost everything to a few experts and the rest receive no gradient. Debug: log the token count per expert per step; a distribution far from uniform confirms collapse.
- Token dropping at capacity. Tokens routed to a full expert are skipped and their representations lose that layer's contribution. Debug: count dropped tokens per batch and correlate with loss spikes.
- Router and expert dtype mismatch. Router scores computed in low precision flip the top-k choice between steps. Debug: compare selected expert ids across two identical forward passes.
In production
Monitor per-expert load, dropped-token rate and all-to-all time as separate metrics. Size memory for total parameters, not activated ones, and cap batch size from expert-parallel communication rather than from local compute.
Common misconceptions
People think: A mixture of experts is cheaper to serve because only a few experts run.
Actually: Compute per token falls, but every expert's weights must be resident, so memory and communication often cost more than the compute saved.
People think: More experts always improve quality.
Actually: Capacity only helps if routing stays balanced and each expert sees enough tokens to train; collapsed routing wastes parameters entirely.
Interview angle
Begin with the split between total and activated parameters, then explain the router, the top-k combine and load balancing as the core difficulty. Connect expert parallelism to all-to-all cost. Weak answers describe MoE as 'using only part of the network' without saying how routing is learned or why balance matters.
Sources
- Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — the sparsely-gated MoE layer and load balancing.
- Fedus, Zoph & Shazeer, Switch Transformers — top-1 routing, capacity factors and training stability.
- Lepikhin et al., GShard — expert parallelism and sharding at scale.