Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 11 of 38

Model parallelism

0 of 38 complete

Why it matters

A model that exceeds one device has to be partitioned, and every partition boundary becomes a place where activations must be exchanged. That exchange costs latency no kernel optimisation can hide, and the cost depends on the interconnect: high-bandwidth links inside a node, much slower links between nodes. The design question is therefore not whether to split but where, and how much communication the latency budget can absorb.

Mental model

It is a group of cooks each responsible for part of a recipe, passing the pan along. The analogy breaks because the hand-off is not free: each pass costs time proportional to the size of the pan and the speed of the kitchen's pass-through window.

How it works

Mechanism, step by stepInteractive
  1. 1Check whether the model fits on one device at the precision and context you need; if it does, prefer replicas.
  2. 2Use tensor parallelism within a node first, where the interconnect is fastest, and keep the group small.
  3. 3Add pipeline parallelism across nodes only when tensor groups cannot span the model, and schedule micro-batches to fill the bubbles.
  4. 4Account for communication in the latency model, including collective operations per layer and their dependence on message size.
  5. 5Keep attention heads and key-value heads divisible by the tensor-parallel degree, or the split needs padding and replication.
  6. 6Benchmark end-to-end request latency at the target concurrency, not per-layer throughput on a single batch.

Step 1 of 6

Check whether the model fits on one device at the precision and context you need; if it does, prefer replicas.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Check whether the model fits on one device at the precision and context you need; if it d… 2. Use tensor parallelism within a node first, where the interconnect is fastest, and keep t… 3. Add pipeline parallelism across nodes only when tensor groups cannot span the model, and … 4. Account for communication in the latency model, including collective operations per layer… 5. Keep attention heads and key-value heads divisible by the tensor-parallel degree, or the … 6. Benchmark end-to-end request latency at the target concurrency, not per-layer throughput …

  1. Check whether the model fits on one device at the precision and context you need; if it does, prefer replicas.
  2. Use tensor parallelism within a node first, where the interconnect is fastest, and keep the group small.
  3. Add pipeline parallelism across nodes only when tensor groups cannot span the model, and schedule micro-batches to fill the bubbles.
  4. Account for communication in the latency model, including collective operations per layer and their dependence on message size.
  5. Keep attention heads and key-value heads divisible by the tensor-parallel degree, or the split needs padding and replication.
  6. Benchmark end-to-end request latency at the target concurrency, not per-layer throughput on a single batch.

Trade-offs

ChoiceGood forCostFails when
Tensor parallel in a nodeLow communication latency, fits large modelsSynchronisation on every layerThe group spans a slow link
Pipeline across nodesScales beyond one nodeBubbles and stage imbalanceTraffic is latency-sensitive and small
Sequence parallelLong contexts without splitting weightsComplex attention communicationContexts are short
Replicas instead of splittingNo communication cost, simple scalingEach replica needs the full model in memoryThe model does not fit on one device
Trade-off mapInteractive

Good for

Low communication latency, fits large models

Cost

Synchronisation on every layer

Fails when

The group spans a slow link

  1. The group spans a slow link
  2. Traffic is latency-sensitive and small
  3. Contexts are short
  4. The model does not fit on one device

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Tensor parallel in a node is good for Low communication latency, fits large models, costs Synchronisation on every layer, and fails when The group spans a slow link. Pipeline across nodes is good for Scales beyond one node, costs Bubbles and stage imbalance, and fails when Traffic is latency-sensitive and small. Sequence parallel is good for Long contexts without splitting weights, costs Complex attention communication, and fails when Contexts are short. Replicas instead of splitting is good for No communication cost, simple scaling, costs Each replica needs the full model in memory, and fails when The model does not fit on one device

Failure modes

  • Tensor group across a slow link. Per-token latency is dominated by all-reduce and throughput collapses. Debug: compare step time with the group inside one node and across nodes.
  • Pipeline bubbles. Devices idle while waiting for the stage ahead, and utilisation looks fine because it is averaged. Debug: measure per-stage idle time over a request batch.
  • Indivisible heads. The parallel degree does not divide the attention heads and the framework silently pads or replicates. Debug: print the shard shapes and compare parameter counts across ranks.

In production

Track per-rank step time and collective duration; an imbalance between ranks shows up as one slow shard holding the whole request. Rehearse losing a rank, because a single failed participant usually fails the entire parallel group.

Common misconceptions

People think: More parallelism always means faster inference.

Actually: Beyond the point where the model fits, added parallelism buys latency, not speed; communication grows with the degree.

People think: Pipeline parallelism is transparent to latency.

Actually: It adds stage hand-offs and bubbles, which is why it suits throughput-oriented batch work more than interactive traffic.

Interview angle

Start from whether the model fits, then choose the split that keeps communication local. A strong answer quantifies the collective cost per layer and explains why pipeline stages need micro-batches. Weak answers list parallelism names without saying what each one costs on the critical path.

Sources