Why it matters
A long document, a broad query or a multi-source question cannot be processed in one call within the latency budget. Splitting it lets branches run concurrently, but the system now has partial failures, a merge step and a burst of requests that can exceed a per-minute limit. The pattern exists to make that parallelism explicit and bounded rather than emergent.
Mental model
A team of readers each taking a chapter, then one editor merging notes into a single summary. The metaphor breaks because readers finish at different times and some do not finish, so the editor needs a rule for missing chapters and the deadline is the editor's, not theirs.
How it works
One split, a width cap, four branches and one join that must decide what a missing branch means.
Step 1 of 6
Choose the split
Branches must be independent and bounded in input size. Parallelism only reduces wall clock when the slowest branch is faster than the serial path and capacity exists.
service
Split + cap
Splitter
independent units
Splits the input into units with bounded size and no dependency between them. If one branch needs another branch's output, the split is wrong and the parallelism is fake.
3 connections
Figure. The topology guarantees that branches are independent, width is capped before launch, and the join sees a completion id for every branch, so a missing result can be detected rather than silently merged. A width of four is an example, not a threshold. It does not guarantee that fan-out lowers latency, that the provider has headroom for the burst, or that a returned branch is cancelled: those depend on the serial baseline, the rate limit and the orphan policy.
What to notice. The join is the design. The red edge into it is a branch timeout, and the red edge back out is an orphan that keeps running after the parent returned. The rate-limit failure at the cap is the third red path, because a burst that trips the tenant limit makes every branch retry.
Text version
A fan-out and fan-in topology in four zones. A request is split into independent units, passes a width cap of four, and launches four branches, each with its own idempotency key, timeout and trace span. All four return to a join that applies a wait-all, quorum or deadline policy, merges deterministically, and returns either a complete or marked partial result. Parent and branch completions are written to a trace and usage store. Red dashed failure edges show branch 3 timing out, the join returning while branch 3 is still running as an orphan, and the width cap tripping a rate limit that makes every branch retry. Six steps cover splitting, capping, fanning out, fanning in, handling a missing branch, and bounding the orphan.
- Decide the split unit so branches are independent and each has a bounded input size.
- Cap the width per request and use a queue or semaphore, so fan-out cannot exceed the provider's rate limit.
- Define the join policy before coding: wait for all, wait for a quorum, or accept partial results with a marker.
- Give each branch a timeout shorter than the overall deadline, and record which branches completed.
- Make the merge deterministic in ordering, since completion order varies between runs.
- Trace the parent and each branch under one id, and attribute tokens per branch.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Wait for all branches | Complete result | One slow branch sets latency | Branches have heavy tails |
| Quorum | Bounded latency | Incomplete evidence | The task needs every source |
| Fixed width | Predictable load | Underuses capacity | Documents are much larger than the cap |
| Dynamic width | Adapts to input | Rate-limit spikes | The provider throttles per minute |
Good for
Complete result
Cost
One slow branch sets latency
Fails when
Branches have heavy tails
- Branches have heavy tails
- The task needs every source
- Documents are much larger than the cap
- The provider throttles per minute
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Wait for all branches is good for Complete result, costs One slow branch sets latency, and fails when Branches have heavy tails. Quorum is good for Bounded latency, costs Incomplete evidence, and fails when The task needs every source. Fixed width is good for Predictable load, costs Underuses capacity, and fails when Documents are much larger than the cap. Dynamic width is good for Adapts to input, costs Rate-limit spikes, and fails when The provider throttles per minute
Failure modes
- Orphaned branch. The parent times out and returns while branches keep running and spending. Debug: compare branch start counts with completion counts for one request id.
- Join without a policy. The merge runs with half the results and the answer silently loses a source. Debug: log completed branch ids and assert the expected count before merging.
- Rate-limit amplification. A burst of fan-out requests trips the tenant limit and every branch retries. Debug: graph request rate and throttling rate per minute during the burst.
In production
Cap width and total branches per request, alert on partial joins, and keep the merge tolerant of a missing branch. Load-test at the rate limit, not below it.
Common misconceptions
People think: Parallel branches always reduce latency.
Actually: They reduce wall clock only when the slowest branch is faster than the serial path and capacity exists.
People think: Fan-in is just concatenation.
Actually: The join decides deduplication, ordering and what to do with missing results, which is most of the design.
Interview angle
State the split, the width cap and the join policy, then the failure behaviour when a branch dies. Strong answers discuss rate limits and partial results. Weak ones draw parallel arrows and never say what happens when one arrow fails.
Sources
- Microsoft, Fan-Out/Fan-In Pattern Scenarios in Durable Functions — parallel branches and the aggregation step.
- Dean & Ghemawat, MapReduce: Simplified Data Processing on Large Clusters — the canonical split, process and merge model.
- Azure Architecture Center, Asynchronous Request-Reply — handling long-running parallel work behind a request.