Why it matters
A single completion can take seconds, and almost all of that time is spent waiting on a socket. Threads make waiting expensive — each one costs stack memory and scheduler attention. Async makes waiting nearly free: one thread runs an event loop that holds many suspended operations and resumes each when its data arrives.
That efficiency is also a hazard. When concurrency is cheap, it is easy to create it accidentally: a fan-out over twenty documents, a retry layer, a background summary, an evaluation replay. Each is reasonable alone. Together they multiply, and the provider sees a request pattern that looks like an attack. Async does not manage load — it exposes how much load you are willing to create.
The second half of the problem is stopping. Users close tabs, navigate away, and hit escape. Without cancellation, the system finishes every abandoned generation, pays for it, writes it to a log nobody reads, and reports latency that excludes the most expensive requests.
Mental model
One barista taking many orders. Start the espresso, take the next order while it brews, steam milk when the machine beeps. One pair of hands, many drinks in progress. Throughput comes from never standing still during a wait.
The analogy breaks in a specific way: a barista can be interrupted between any two movements, while a coroutine only yields at an await. One CPU-bound or blocking call holds the entire queue, and every other in-flight request waits behind it. There is no preemption to save you.
How it works
One thread, one loop. Every arrow below the request is in flight at the same time; the stall is the only interval in which nothing can be delivered.
Client
deadline 5 s
Event loop
one thread
Provider
model calls
{ "doc": "refund policy",
"score": 0.91 }
120 ms. The task resumes, emits a token,
then awaits the next read. The other seven
are still suspended.Step 1: fan-out. Hover, tap or tab to any message to inspect its payload.
Figure. Concurrency overlaps waiting, not work. Eight retrievals are in flight when a six-megabyte parse runs for 1.2 s without an await, so every ready task and every timer waits behind it: the deadline fires late, the cancel reaches the provider late, and the client receives a timeout after the budget is already gone.
What to notice. The failure is a blocking call on the loop: timers, cancellation and every suspended task are served only at an await, so one synchronous parse turns a 5 s budget into a 5.1 s response. CPU work belongs in an executor. Client-disconnect cancellation takes the same path and is not drawn separately; timings and ids are illustrative.
Text version
Sequence across Client, Event loop and Provider. Step 1, fan-out: Client to Event loop GET /answer with a 5000 ms request deadline; Event loop to Provider eight retrieval calls under a Semaphore of 8 with a 4 s per-attempt timeout, created in a TaskGroup; Provider to Event loop the first result after 120 ms. Step 2, stall: an in-process Event loop row runs json.loads, normalise and rerank for 1.2 s with no await, so every ready task and timer waits. Step 3, late cancel: Event loop to Provider cancel_remaining, delivered at each task's next await, closing the upstream sockets 100 ms after the deadline was due. Step 4, deadline: Event loop to Client HTTP 504 with error deadline_exceeded, elapsed_ms 5100 and budget_ms 5000, because the budget check itself ran after the stall. Step 5, measure: an in-process Event loop row with event_loop_lag_ms 1180, in_flight 8, queue_depth 0 and cancellations 7; lag tracks request rate rather than work done, and the parse must move to an executor.
- The loop runs tasks and switches at
await. A coroutine is a suspendable function. When it awaits I/O, it yields control; the loop runs another task; when the I/O completes, the first resumes. Nothing runs in parallel on that thread. - Create tasks deliberately. A bare coroutine call does nothing until awaited or scheduled. Fan-out means creating tasks, and each task is a unit you are responsible for awaiting or cancelling.
- Bound the concurrency. A semaphore around outbound calls is the cheapest protection against self-inflicted overload. Size it from the downstream limit, not from optimism.
- Apply two levels of timeout. A per-attempt timeout stops a single stuck call, and a request deadline stops the aggregate. Ten calls in parallel with a thirty-second per-attempt timeout can still exhaust a thirty-second user budget; a deadline caps the aggregate, and a semaphore bounds how many run at once.
- Structure the fan-out. A Python
TaskGrouppropagates failure and cancels its siblings on exit, so no task outlives the scope. JavaScript has no equivalent:Promise.allrejects on the first error but leaves the other promises running. Pair it with a sharedAbortController, abort it in the catch, and pass the signal into every request. - Propagate cancellation. Catch the cancellation signal, clean up, and re-raise. In Python,
CancelledErrorderives fromBaseExceptionprecisely so thatexcept Exceptiondoes not swallow it; catchingBaseExceptionor a bareexceptdoes. In the browser, pass anAbortSignaldown tofetchso the socket closes. - Apply backpressure. A bounded queue that blocks or rejects when full is a feature. An unbounded queue converts a spike into memory growth and then into a timeout storm.
- Drain on shutdown. Stop accepting, let in-flight work finish or cancel it explicitly, then close. Killing the process mid-write is how you get half-written records.
- Good fit
- Model calls, HTTP, database reads, streaming
- Bad fit
- CPU-heavy preprocessing on the event loop
- Python primitives
- asyncio.TaskGroup, asyncio.timeout, Semaphore
- JS primitives
- Promise.allSettled, AbortSignal, bounded queues
- First symptom of a blocked loop
- Latency proportional to total concurrency, not work done
A bounded fan-out
import asyncio
async def fetch_all(urls: list[str], limit: int = 8, deadline: float = 20.0):
sem = asyncio.Semaphore(limit)
async def one(url: str):
async with sem: # cap total in-flight calls
async with asyncio.timeout(deadline): # cap each attempt
return await client.get(url)
async with asyncio.TaskGroup() as group: # any failure cancels siblings
tasks = [group.create_task(one(u)) for u in urls]
return [t.result() for t in tasks] # order preserved by constructionasyncio.TaskGroup and asyncio.timeout require Python 3.11 or newer. On older versions, gather(return_exceptions=False) plus wait_for covers most of it, but you lose the automatic sibling cancellation.
Trade-offs
| Concurrency model | Good for | Cost | Fails when |
|---|---|---|---|
| Single event loop, async I/O | Many concurrent model and HTTP calls | One blocking call stalls everything | Libraries are sync-only or work is CPU-bound |
| Thread pool | Blocking SDKs, moderate concurrency | Memory per thread, context switching | Concurrency reaches hundreds per process |
| Process pool | CPU-bound preprocessing, tokenisation | Serialisation overhead, no shared state | The work is I/O and the payloads are large |
| Horizontal replicas | Isolation and simple scaling | Cost, cold starts, duplicated caches | The bottleneck is a shared downstream quota |
| Unbounded fan-out | Maximum throughput on an idle system | Retry storms, timeouts, provider throttling | Any dependency degrades |
Good for
Many concurrent model and HTTP calls
Cost
One blocking call stalls everything
Fails when
Libraries are sync-only or work is CPU-bound
- Libraries are sync-only or work is CPU-bound
- Concurrency reaches hundreds per process
- The work is I/O and the payloads are large
- The bottleneck is a shared downstream quota
- Any dependency degrades
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Single event loop, async I/O is good for Many concurrent model and HTTP calls, costs One blocking call stalls everything, and fails when Libraries are sync-only or work is CPU-bound. Thread pool is good for Blocking SDKs, moderate concurrency, costs Memory per thread, context switching, and fails when Concurrency reaches hundreds per process. Process pool is good for CPU-bound preprocessing, tokenisation, costs Serialisation overhead, no shared state, and fails when The work is I/O and the payloads are large. Horizontal replicas is good for Isolation and simple scaling, costs Cost, cold starts, duplicated caches, and fails when The bottleneck is a shared downstream quota. Unbounded fan-out is good for Maximum throughput on an idle system, costs Retry storms, timeouts, provider throttling, and fails when Any dependency degrades
Failure modes
A blocking call on the loop. One synchronous client or a large JSON parse serialises every request. Debug: measure event-loop lag; if p99 lag tracks request rate, find the sync call by sampling stacks under load.
Unbounded fan-out. One user action produces a burst of provider calls, which returns 429s, which are retried, which produces more calls. Debug: count outbound requests per inbound request in a trace and compare with the cap you intended.
Swallowed cancellation. Cleanup code catches everything, the task keeps running, and the client sees an error while the server keeps billing. Debug: cancel a request in staging and assert the upstream call is closed within the deadline.
Fire-and-forget tasks. A background task is created without a reference; if it fails, nothing notices, and on shutdown it is killed mid-write. Debug: require every task to be owned by a scope, and alert on tasks still running past their deadline.
In production
The metrics that matter are event-loop lag, in-flight requests, queue depth, per-dependency concurrency and cancellation rate. A concurrency limit per provider is a cost control as much as a reliability control: it caps the blast radius of a retry loop. On shutdown, stop accepting new work, give in-flight requests a deadline, and cancel the rest explicitly so partial writes are visible rather than silent. And keep the model call inside the same cancellation scope as the request — a generation that outlives its client is pure cost.
Common misconceptions
People think: Async code runs in parallel.
Actually: It overlaps waiting on one thread. Two coroutines never execute CPU instructions at the same time, and a synchronous call blocks every other task.
People think: Cancelling a task frees the resources immediately.
Actually: Cancellation is a request delivered at the next await. Code that ignores it, or blocks without awaiting, keeps running. Upstream sockets close only if you propagate the signal.
People think: More concurrency means more throughput.
Actually: Throughput rises until the first saturated dependency, then falls as queues, timeouts and retries consume capacity. The useful number is the concurrency your slowest dependency can absorb.
Interview angle
Expect "how would you fan out retrieval across twenty documents?" and then "what happens when the provider slows down?" The second question separates people who have run this in production. Answer with the cap, the deadline, the cancellation path and the metric — and mention that the queue depth, not the average latency, is the early warning.
See production questions and retries, timeouts and jitter.
Sources
- Python
asynciotasks and coroutines — task groups, timeouts, cancellation semantics and the rules for awaiting tasks. - Python
asyncioqueues — bounded queues and the primitives for backpressure between producers and consumers. - MDN:
AbortController— the browser-side cancellation signal that closes an in-flight request.