Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 14 of 38

Autoscaling

0 of 38 complete

Why it matters

A web service scales on request rate because a new process is ready in seconds. An inference replica must load weights, allocate a cache pool and warm kernels before it can serve, so the signal has to arrive earlier and the reaction has a floor. Scaling on utilisation is especially misleading, because a device can sit at high utilisation while requests queue, and low utilisation while a few long generations hold cache.

Mental model

It is staffing a kitchen by watching the queue at the counter rather than the heat of the stove. The analogy breaks because a new cook needs an hour of training before touching an order, so the queue you see now is not the queue you will be able to serve.

How it works

Autoscaling: arrivals, replicas, queueInteractive

Phase 1 of 5baseline

Step 1/5 · baseline
060120180240time (s)ARRIVALS (REQ/S)READY REPLICASQUEUE DEPTH (REQUESTS)02501205001kcold start40 s
arrivalsready replicasrequestedcold start

Model output · synthetic, rounded

Arrivals
15/sfrom 5/s
Ready replicas
2 → 7ready at ≈85 s
Peak queue
≈410requestsbefore the new replicas
Queue empty
≈155sdrains 6/s

Model output

Baseline: 5 req/s arriving against 2 ready replicas, 6 req/s of capacity. The queue stays at zero.

Illustrative model

Assumption: 3 req/s per replica at an 80% target policy (integer replicas overshoot). Real capacity depends on model, context, batching and hardware.

Also fixed: constant arrivals, a 60-request queue-depth threshold and a 40 s cold start. Scale-in and flapping are not modelled.

A real burst is uneven, readiness waits on scheduling, image pull, weights and warmup, and the queue is per model. None of those details are in this curve.

Figure. Three aligned panels share one time axis: arrivals per second, ready replicas and queue depth. The autoscaler scales on queue depth crossing 60 requests — a proxy for wait — and the shaded gap is the 40 s cold start after that decision. The model is illustrative, not measured.

What to notice. The drawn policy scales on queue depth, a proxy for wait, so it fires while the queue is growing, not after device utilisation saturates. A wait or time-to-first-token signal can react earlier still, but no signal can serve during the cold-start gap — which is why a floor of warm replicas exists.

Text version

Three stacked panels share a time axis from 0 to 240 seconds. Arrivals (top) are bars at 5 requests per second until 40 seconds, then 15. Ready replicas (middle) hold a step line at 2 until about 85 seconds, then 7; a dashed requested line rises to 7 at 46 seconds. Queue depth (bottom) is an area that stays at zero, grows through the shaded 40-second cold-start gap to about 410 requests, then drains to zero at about 155 seconds. The autoscaler scales on queue depth, not device utilisation. The burst, the times and the queue are model outputs, not measurements.

  1. Choose a signal that moves before users complain: pending requests, queue wait, or time-to-first-token against its objective.
  2. Measure cold-start time end to end, including scheduling, image pull, weight load and a warmup generation.
  3. Set a minimum replica count from the burst you must absorb within one cold-start window.
  4. Scale out aggressively on the leading signal and scale in on a slower average with a stabilisation window.
  5. Drain before terminating, so in-flight generations finish and cache is released cleanly.
  6. Load-test a synthetic burst and record how long the system stays above its latency objective.

Trade-offs

ChoiceGood forCostFails when
Scale on queue waitTracks user-visible delayNoisy under bursty trafficTraffic is uniform and predictable
Scale on device utilisationSimple metric, widely availableSaturation lags the queueLong generations hold memory without compute
Aggressive scale-inLower idle costDiscards warm cache and thrashesArrivals are bursty
High minimum replicasAbsorbs bursts immediatelyPays for idle capacityLoad is genuinely intermittent
Trade-off mapInteractive

Good for

Tracks user-visible delay

Cost

Noisy under bursty traffic

Fails when

Traffic is uniform and predictable

  1. Traffic is uniform and predictable
  2. Long generations hold memory without compute
  3. Arrivals are bursty
  4. Load is genuinely intermittent

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Scale on queue wait is good for Tracks user-visible delay, costs Noisy under bursty traffic, and fails when Traffic is uniform and predictable. Scale on device utilisation is good for Simple metric, widely available, costs Saturation lags the queue, and fails when Long generations hold memory without compute. Aggressive scale-in is good for Lower idle cost, costs Discards warm cache and thrashes, and fails when Arrivals are bursty. High minimum replicas is good for Absorbs bursts immediately, costs Pays for idle capacity, and fails when Load is genuinely intermittent

Failure modes

  • Flapping. Replicas are added and removed every few minutes and each new one pays cold start. Debug: plot replica count and cold-start count over an hour and look for alternation.
  • Scale-out slower than the burst. The queue grows faster than replicas become ready and the objective is missed for the whole burst. Debug: compare burst arrival rate with replica readiness rate from the event log.
  • Utilisation-triggered starvation. The autoscaler adds replicas on high utilisation, but the real limit is cache per replica. Debug: correlate queue wait with cache utilisation rather than compute utilisation.

In production

Alert on time above the latency objective, not on replica count. Keep a floor of warm capacity for the daily peak and review cold-start time whenever the model or image changes.

Common misconceptions

People think: Autoscaling removes the need for capacity planning.

Actually: It moves the problem to cold start and burst absorption; the floor and the ramp rate still have to be designed.

People think: Device utilisation is the right scaling signal.

Actually: Utilisation can be high while requests queue and low while memory is exhausted; queue-derived signals track user experience better.

Interview angle

Pick a leading signal, measure cold start, and explain the floor and the asymmetric scale-out and scale-in. Name the failure of scaling on utilisation. Saying it scales with load, with the reaction time unexamined, is the gap.

Sources