Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 31 of 33

MoE

0 of 33 complete

Why it matters

Dense scaling raises compute and memory together, but quality tends to improve with parameters while serving cost tracks compute. A mixture of experts breaks that coupling: many more parameters are stored, yet each token is processed by only a couple of small networks. The price is a routing decision that must be learned, balanced and made deterministic enough for distributed training and batched inference.

Mental model

It is a hospital with many specialist departments and a triage desk: each patient sees two specialists, so the building is large but each visit is short. The metaphor breaks because triage is learned and differentiable only through the router's scores, and a popular department can be overwhelmed while others sit idle.

How it works

Router · eight tokens over five expertsInteractive

hover or focus a cell

00.700
token ↓expert →
E1E2E3E4E5Σ
t11.000
t21.000
t31.000
t41.000
t51.000
t61.000
t71.000
t81.000
mass2.1403.3301.1300.6900.710

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

top-k
2 of 5
slots filled
8 × 2 = 16
even share
16 ÷ 5 = 3.2
expert 4 slots
0 of 16

Step 1 of 4

One row per token

The router scores every expert for every token before anything is dropped. Each row is a distribution over all five experts, so a low cell is still a considered score, not an absent one.

Next: Top two for t3

Step 1/4 · One row per token

Figure. One mixture-of-experts layer with eight tokens and five experts, illustrative router probabilities. Each row is a token's softmax over every expert and totals 1.000; the top two cells in each row are the experts that run for that token. Column totals add the printed probabilities, so they measure routing mass rather than the number of tokens assigned.

What to notice. The top-two rule concentrates this batch: expert 2 takes 6 of the 16 slots, expert 1 takes 5, expert 5 takes 3, expert 3 takes 2 and expert 4 takes none, against 16 ÷ 5 = 3.2 slots if routing were even. Expert 4 also has the smallest column total, 0.690, so the imbalance is visible before any token is dropped at capacity. The probabilities are invented; the slot counts follow the rows exactly, and some implementations renormalise the two selected scores while others combine with the raw probabilities.

Text version

An eight by five matrix. Rows are tokens t1 to t8 and columns experts E1 to E5. Each cell prints a router probability and each row total reads 1.000. Row t1 prints 0.640, 0.100, 0.080, 0.060 and 0.120; t2 prints 0.580, 0.120, 0.090, 0.070 and 0.140; t3 prints 0.140, 0.520, 0.100, 0.080 and 0.160; t4 prints 0.160, 0.600, 0.110, 0.080 and 0.050; t5 prints 0.180, 0.550, 0.130, 0.090 and 0.050; t6 prints 0.120, 0.500, 0.220, 0.100 and 0.060; t7 prints 0.140, 0.480, 0.240, 0.090 and 0.050; t8 prints 0.180, 0.460, 0.160, 0.120 and 0.080. Column totals read 2.140, 3.330, 1.130, 0.690 and 0.710. Steps highlight the two selected cells for t3, the E2 and E4 columns, and the six E2 cells that show the load concentration.

  1. Replace the dense feed-forward block with a set of expert networks, each with its own parameters but the same shape.
  2. Compute a router score per token per expert, usually a linear projection followed by a softmax over experts.
  3. Select the top-k experts per token and run only those experts on that token.
  4. Combine the expert outputs weighted by their router probabilities, so the result stays differentiable with respect to routing.
  5. Apply a capacity factor and a load-balancing loss so no expert receives far more tokens than the average.
  6. All-reduce across devices when experts are sharded, since a token's chosen expert may live on another rank.

Trade-offs

ChoiceGood forCostFails when
Top-1 routingCheapest per token, simple combineCoarse specialisation, brittle to router noiseTokens need more than one kind of transformation
Top-2 routingBetter quality, smoother gradientsDouble the expert compute per tokenLatency budget is tight
Expert parallelismModels far larger than one deviceAll-to-all communication dominatesInterconnect is slow or batches are small
Large capacity factorFewer dropped tokensMore padding and wasted computeTraffic is skewed across experts
Trade-off mapInteractive

Good for

Cheapest per token, simple combine

Cost

Coarse specialisation, brittle to router noise

Fails when

Tokens need more than one kind of transformation

  1. Tokens need more than one kind of transformation
  2. Latency budget is tight
  3. Interconnect is slow or batches are small
  4. Traffic is skewed across experts

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Top-1 routing is good for Cheapest per token, simple combine, costs Coarse specialisation, brittle to router noise, and fails when Tokens need more than one kind of transformation. Top-2 routing is good for Better quality, smoother gradients, costs Double the expert compute per token, and fails when Latency budget is tight. Expert parallelism is good for Models far larger than one device, costs All-to-all communication dominates, and fails when Interconnect is slow or batches are small. Large capacity factor is good for Fewer dropped tokens, costs More padding and wasted compute, and fails when Traffic is skewed across experts

Failure modes

  • Expert collapse. The router sends almost everything to a few experts and the rest receive no gradient. Debug: log the token count per expert per step; a distribution far from uniform confirms collapse.
  • Token dropping at capacity. Tokens routed to a full expert are skipped and their representations lose that layer's contribution. Debug: count dropped tokens per batch and correlate with loss spikes.
  • Router and expert dtype mismatch. Router scores computed in low precision flip the top-k choice between steps. Debug: compare selected expert ids across two identical forward passes.

In production

Monitor per-expert load, dropped-token rate and all-to-all time as separate metrics. Size memory for total parameters, not activated ones, and cap batch size from expert-parallel communication rather than from local compute.

Common misconceptions

People think: A mixture of experts is cheaper to serve because only a few experts run.

Actually: Compute per token falls, but every expert's weights must be resident, so memory and communication often cost more than the compute saved.

People think: More experts always improve quality.

Actually: Capacity only helps if routing stays balanced and each expert sees enough tokens to train; collapsed routing wastes parameters entirely.

Interview angle

Begin with the split between total and activated parameters, then explain the router, the top-k combine and load balancing as the core difficulty. Connect expert parallelism to all-to-all cost. Weak answers describe MoE as 'using only part of the network' without saying how routing is learned or why balance matters.

Sources