Why it matters
Every service has a capacity, and demand that approaches it produces queueing delay no amount of optimisation removes. For inference the effect is sharper because service time varies with output length: a few long generations occupy slots and raise wait time for everyone behind them. Teams that only measure average throughput miss the transition, then meet it during a launch when retries add load on top of the original demand.
Mental model
It is a checkout line: the cashier's speed sets throughput, and the line length sets your wait. The analogy breaks because customers in an inference queue can be told to come back later, and the queue itself can be shortened by rejecting work.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Time in system and queue depth against offered load for a 150 ms service time and a 100 requests-per-second capacity. The curves follow the single-server queueing shape; the constants are illustrative and the divergence at saturation is the claim.
What to notice. At 50 requests a second a request spends 300 ms in the system, half of it waiting. At 90 it spends 1,500 ms and 1,350 ms of that is queue. The 100 requests a second line is the service rate: as load approaches it the wait diverges, the drawn curves stop at 95 req/s where the wait is already 3 s, and only admission control changes the outcome.
Text version
Two stacked panels share an x axis of offered load from 0 to 140 requests a second. The upper panel plots time per request, rising from 150 ms at zero load to 3,000 ms at 95, with a flat dashed service-time line at 150 ms beneath it, a vertical capacity line at 100, and a knee marker at 85 requests a second reading 1,000 ms. The lower panel plots queue depth rising from 0 to 18 requests over the same range, with a horizontal example queue bound at 15 that the curve crosses near 94. A slider moves a cursor along the axis and reads time and queue depth together; beyond 95 requests a second the readout reports off chart. Three step controls reveal the service time, then the wait, then the capacity line and knee.
- Measure arrival rate and the service-time distribution, including the long tail of output lengths.
- Compute the concurrency implied by the latency objective, using the relationship between arrivals, time in system and requests in flight.
- Set a queue bound and a per-request deadline, so waiting is capped before work starts.
- Add admission control that rejects or defers when the bound is reached, with a retry hint the client can honour.
- Separate interactive and batch traffic into different pools so batch work cannot occupy interactive slots.
- Load-test above capacity and confirm the system fails fast rather than slowly.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Unbounded queue | No rejected requests | Latency grows without limit and timeouts cascade | Arrivals exceed capacity for more than a moment |
| Bounded queue with rejection | Predictable latency, clear signal | Some requests fail | Clients retry without backoff |
| Priority pools | Protects interactive traffic | Batch work starves under load | All traffic has the same deadline |
| Over-provisioning | Absorbs bursts | Idle cost | Load is spiky and mostly low |
Good for
No rejected requests
Cost
Latency grows without limit and timeouts cascade
Fails when
Arrivals exceed capacity for more than a moment
- Arrivals exceed capacity for more than a moment
- Clients retry without backoff
- All traffic has the same deadline
- Load is spiky and mostly low
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Unbounded queue is good for No rejected requests, costs Latency grows without limit and timeouts cascade, and fails when Arrivals exceed capacity for more than a moment. Bounded queue with rejection is good for Predictable latency, clear signal, costs Some requests fail, and fails when Clients retry without backoff. Priority pools is good for Protects interactive traffic, costs Batch work starves under load, and fails when All traffic has the same deadline. Over-provisioning is good for Absorbs bursts, costs Idle cost, and fails when Load is spiky and mostly low
Failure modes
- Queue collapse. Wait time exceeds client timeouts, clients retry, and effective arrival rate doubles. Debug: compare server-side arrival rate with client-side request count for the same window.
- Head-of-line blocking. Long generations occupy slots and short requests wait behind them. Debug: plot wait time by requested output length.
- Average hides the knee. Mean latency looks acceptable while p95 has already left the objective. Debug: chart p50, p95 and p99 against utilisation on the same axis.
In production
Alert on queue wait and queue depth rather than on throughput, and cap the queue so overload produces rejections you can count. Keep a load-shedding path tested at the same time as the happy path.
Common misconceptions
People think: More throughput always improves the user experience.
Actually: Throughput without admission control can raise latency for everyone; the objective is served requests within a deadline.
People think: Queues smooth out traffic spikes.
Actually: A queue absorbs a short spike and amplifies a sustained one, because waiting work holds memory and invites retries.
Interview angle
Use the relationship between arrivals, concurrency and time in system to show why latency explodes near capacity, then describe admission control and deadlines. A strong answer names the queue bound and the rejection signal. Weak answers promise to optimise the model and never discuss the queue.
Sources
- Dean and Barroso, The Tail at Scale — why tail latency dominates at scale and how queues amplify it.
- Little's law — the relationship between arrivals, time in system and concurrency.
- AWS Builders' Library, Using load shedding to avoid overload — admission control as an overload strategy.