Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 12 of 38

Distributed inference

0 of 38 complete

Why it matters

Once a model is split across devices, a single user request touches several processes, and the front end must choose a replica group, keep its cache warm and decide what to do when one member is unhealthy. Naive load balancing spreads requests evenly and destroys prefix reuse; naive retries multiply work across every shard. The problem is preserving locality and bounding blast radius while still using the whole cluster.

Mental model

It is a kitchen with stations: one preps, one cooks, one plates, and the ticket is done only when all three finish. The analogy breaks at routing, because a good expediter sends similar tickets to the same station to reuse what is already prepped.

How it works

One request across four ranks, then a retryInteractive
workprefilldecoderetrywaitretryunlinked workevent
20 ms

Event · step 1 of 6

Route and prefill

At 20 ms

The front end hashes the prompt prefix and sends the request to a four-rank replica group; a round-robin router would throw that locality away. The request spans all four ranks, and each processes the prompt. Every prefill is still running; rank 2 will be the last to finish.

On screen at this event

  • route
  • prefill
  • prefill
  • prefill · slow
  • prefill

Next at 100 ms: Decode begins on three ranks

End-to-end
—
Slowest rank
rank 2
Retried work
0
Uncancelled
0
Wasted rank-ms
0
Step 1/6 · Route and prefill

Figure. A synthetic 380 ms request on a four-rank group: prefill, decode, a 250 ms timeout and a whole-request retry on the same ranks. Rank 2's original work continues after the retry has started. Timings are illustrative and the cache state is not shown.

What to notice. A request is only as fast as its slowest rank. Rank 2's prefill sets the tail, and the retry repeats prefill on all four ranks rather than on the one that lagged. The ghost span after 262 ms is the old rank 2 still decoding because cancellation did not propagate: the cluster paid twice and its cache stayed pinned. Per-shard health plus cancel-to-all would remove both costs. Wasted rank-ms counts retry prefill, retry decode and the uncancelled ghost from the spans: 36 at 262 ms, 406 at 336 ms and 578 at 380 ms.

Text version

Six lanes: client, gateway, rank 0 to rank 3. The client waits 0 to 380 ms. The gateway routes from 0 to 12, waits for the ranks until 250, retries at 250, and waits for the retry until 378. Ranks 0, 1 and 3 prefill by 100 ms and decode from about 100 to 250. Rank 2's prefill runs 12 to 262 in ochre and is labelled slow; then a dashed ghost decode runs 262 to 340 labelled ghost. On a second row, all four ranks run retry prefill from about 250 to 336 and retry decode to 378. The retry is a second pass on the same ranks, not a second group. A red marker at 250 reads timeout · retry. Phases are prefill 0–100, decode 100–250 and retry 250–380. Counters: end-to-end 380 ms at the final step; slowest rank rank 2; retried work four ranks; uncancelled one rank; wasted rank-ms about 36 at 262 ms, 406 at 336 ms and 578 at 380 ms, counting retry prefill, retry decode and the uncancelled ghost.

  1. Decide the sharding topology and record which ranks must participate in every request.
  2. Route by prefix affinity where possible, so a repeated system prompt lands on a replica that already holds its blocks.
  3. Consider separate pools for prompt processing and decoding, sized for the compute and memory profile of each phase.
  4. Define health per shard and a policy for a partially healthy group: fail the request, retry elsewhere, or serve a degraded model.
  5. Propagate cancellation to every shard so an abandoned request stops consuming cache.
  6. Measure end-to-end latency percentiles per group, and keep per-shard timings to locate the slow participant.

Trade-offs

ChoiceGood forCostFails when
Round-robin routingEven load and simple implementationDiscards prefix cache localityPrompts share long prefixes
Prefix-affinity routingHigh cache reuseHot spots and uneven loadTraffic has little prefix overlap
Split prefill and decodeEach phase runs on suitable hardwareCache transfer between poolsInterconnect is slow or contexts are short
Replicated small groupsSmaller blast radiusMore groups to operateThe model needs the whole cluster
Trade-off mapInteractive

Good for

Even load and simple implementation

Cost

Discards prefix cache locality

Fails when

Prompts share long prefixes

  1. Prompts share long prefixes
  2. Traffic has little prefix overlap
  3. Interconnect is slow or contexts are short
  4. The model needs the whole cluster

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Round-robin routing is good for Even load and simple implementation, costs Discards prefix cache locality, and fails when Prompts share long prefixes. Prefix-affinity routing is good for High cache reuse, costs Hot spots and uneven load, and fails when Traffic has little prefix overlap. Split prefill and decode is good for Each phase runs on suitable hardware, costs Cache transfer between pools, and fails when Interconnect is slow or contexts are short. Replicated small groups is good for Smaller blast radius, costs More groups to operate, and fails when The model needs the whole cluster

Failure modes

  • Slow shard, slow request. One rank falls behind and every request through its group misses the latency target. Debug: compare per-rank step durations for the same request id.
  • Affinity hot spot. Prefix routing sends most traffic to a few replicas and queues build there. Debug: plot requests per replica against cache reuse per replica.
  • Cancellation not propagated. A closed client connection leaves shards generating and holding blocks. Debug: count active sequences per shard after a burst of aborted requests.

In production

Monitor per-group latency, per-shard error rate and cache reuse by replica. Drain groups rather than killing them during a deploy, and keep headroom so losing one group does not overload the rest.

Common misconceptions

People think: Distributed inference is just more devices on the same endpoint.

Actually: The endpoint becomes a router with locality and partial-failure semantics that a single-device deployment does not have.

People think: Load balancing should always be even.

Actually: Even distribution maximises fairness and minimises cache reuse; the right policy depends on whether prompts share prefixes.

Interview angle

Describe the request path across shards, then the two decisions it forces: locality and partial failure. A strong answer explains prefix affinity and what happens when a rank lags. Weak answers treat it as a bigger machine.

Sources