Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 15 of 38

Throughput & queueing

0 of 38 complete

Why it matters

Every service has a capacity, and demand that approaches it produces queueing delay no amount of optimisation removes. For inference the effect is sharper because service time varies with output length: a few long generations occupy slots and raise wait time for everyone behind them. Teams that only measure average throughput miss the transition, then meet it during a launch when retries add load on top of the original demand.

Mental model

It is a checkout line: the cashier's speed sets throughput, and the line length sets your wait. The analogy breaks because customers in an inference queue can be told to come back later, and the queue itself can be shortened by rejecting work.

How it works

The knee is where the queue sets the timeInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Time in system and queue depth against offered load for a 150 ms service time and a 100 requests-per-second capacity. The curves follow the single-server queueing shape; the constants are illustrative and the divergence at saturation is the claim.

What to notice. At 50 requests a second a request spends 300 ms in the system, half of it waiting. At 90 it spends 1,500 ms and 1,350 ms of that is queue. The 100 requests a second line is the service rate: as load approaches it the wait diverges, the drawn curves stop at 95 req/s where the wait is already 3 s, and only admission control changes the outcome.

Text version

Two stacked panels share an x axis of offered load from 0 to 140 requests a second. The upper panel plots time per request, rising from 150 ms at zero load to 3,000 ms at 95, with a flat dashed service-time line at 150 ms beneath it, a vertical capacity line at 100, and a knee marker at 85 requests a second reading 1,000 ms. The lower panel plots queue depth rising from 0 to 18 requests over the same range, with a horizontal example queue bound at 15 that the curve crosses near 94. A slider moves a cursor along the axis and reads time and queue depth together; beyond 95 requests a second the readout reports off chart. Three step controls reveal the service time, then the wait, then the capacity line and knee.

  1. Measure arrival rate and the service-time distribution, including the long tail of output lengths.
  2. Compute the concurrency implied by the latency objective, using the relationship between arrivals, time in system and requests in flight.
  3. Set a queue bound and a per-request deadline, so waiting is capped before work starts.
  4. Add admission control that rejects or defers when the bound is reached, with a retry hint the client can honour.
  5. Separate interactive and batch traffic into different pools so batch work cannot occupy interactive slots.
  6. Load-test above capacity and confirm the system fails fast rather than slowly.

Trade-offs

ChoiceGood forCostFails when
Unbounded queueNo rejected requestsLatency grows without limit and timeouts cascadeArrivals exceed capacity for more than a moment
Bounded queue with rejectionPredictable latency, clear signalSome requests failClients retry without backoff
Priority poolsProtects interactive trafficBatch work starves under loadAll traffic has the same deadline
Over-provisioningAbsorbs burstsIdle costLoad is spiky and mostly low
Trade-off mapInteractive

Good for

No rejected requests

Cost

Latency grows without limit and timeouts cascade

Fails when

Arrivals exceed capacity for more than a moment

  1. Arrivals exceed capacity for more than a moment
  2. Clients retry without backoff
  3. All traffic has the same deadline
  4. Load is spiky and mostly low

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Unbounded queue is good for No rejected requests, costs Latency grows without limit and timeouts cascade, and fails when Arrivals exceed capacity for more than a moment. Bounded queue with rejection is good for Predictable latency, clear signal, costs Some requests fail, and fails when Clients retry without backoff. Priority pools is good for Protects interactive traffic, costs Batch work starves under load, and fails when All traffic has the same deadline. Over-provisioning is good for Absorbs bursts, costs Idle cost, and fails when Load is spiky and mostly low

Failure modes

  • Queue collapse. Wait time exceeds client timeouts, clients retry, and effective arrival rate doubles. Debug: compare server-side arrival rate with client-side request count for the same window.
  • Head-of-line blocking. Long generations occupy slots and short requests wait behind them. Debug: plot wait time by requested output length.
  • Average hides the knee. Mean latency looks acceptable while p95 has already left the objective. Debug: chart p50, p95 and p99 against utilisation on the same axis.

In production

Alert on queue wait and queue depth rather than on throughput, and cap the queue so overload produces rejections you can count. Keep a load-shedding path tested at the same time as the happy path.

Common misconceptions

People think: More throughput always improves the user experience.

Actually: Throughput without admission control can raise latency for everyone; the objective is served requests within a deadline.

People think: Queues smooth out traffic spikes.

Actually: A queue absorbs a short spike and amplifies a sustained one, because waiting work holds memory and invites retries.

Interview angle

Use the relationship between arrivals, concurrency and time in system to show why latency explodes near capacity, then describe admission control and deadlines. A strong answer names the queue bound and the rejection signal. Weak answers promise to optimise the model and never discuss the queue.

Sources