Why it matters
A web service scales on request rate because a new process is ready in seconds. An inference replica must load weights, allocate a cache pool and warm kernels before it can serve, so the signal has to arrive earlier and the reaction has a floor. Scaling on utilisation is especially misleading, because a device can sit at high utilisation while requests queue, and low utilisation while a few long generations hold cache.
Mental model
It is staffing a kitchen by watching the queue at the counter rather than the heat of the stove. The analogy breaks because a new cook needs an hour of training before touching an order, so the queue you see now is not the queue you will be able to serve.
How it works
Phase 1 of 5baseline
Model output · synthetic, rounded
- Arrivals
- 15/sfrom 5/s
- Ready replicas
- 2 → 7ready at ≈85 s
- Peak queue
- ≈410requestsbefore the new replicas
- Queue empty
- ≈155sdrains 6/s
Model output
Baseline: 5 req/s arriving against 2 ready replicas, 6 req/s of capacity. The queue stays at zero.
Illustrative model
Assumption: 3 req/s per replica at an 80% target policy (integer replicas overshoot). Real capacity depends on model, context, batching and hardware.
Also fixed: constant arrivals, a 60-request queue-depth threshold and a 40 s cold start. Scale-in and flapping are not modelled.
A real burst is uneven, readiness waits on scheduling, image pull, weights and warmup, and the queue is per model. None of those details are in this curve.
Figure. Three aligned panels share one time axis: arrivals per second, ready replicas and queue depth. The autoscaler scales on queue depth crossing 60 requests — a proxy for wait — and the shaded gap is the 40 s cold start after that decision. The model is illustrative, not measured.
What to notice. The drawn policy scales on queue depth, a proxy for wait, so it fires while the queue is growing, not after device utilisation saturates. A wait or time-to-first-token signal can react earlier still, but no signal can serve during the cold-start gap — which is why a floor of warm replicas exists.
Text version
Three stacked panels share a time axis from 0 to 240 seconds. Arrivals (top) are bars at 5 requests per second until 40 seconds, then 15. Ready replicas (middle) hold a step line at 2 until about 85 seconds, then 7; a dashed requested line rises to 7 at 46 seconds. Queue depth (bottom) is an area that stays at zero, grows through the shaded 40-second cold-start gap to about 410 requests, then drains to zero at about 155 seconds. The autoscaler scales on queue depth, not device utilisation. The burst, the times and the queue are model outputs, not measurements.
- Choose a signal that moves before users complain: pending requests, queue wait, or time-to-first-token against its objective.
- Measure cold-start time end to end, including scheduling, image pull, weight load and a warmup generation.
- Set a minimum replica count from the burst you must absorb within one cold-start window.
- Scale out aggressively on the leading signal and scale in on a slower average with a stabilisation window.
- Drain before terminating, so in-flight generations finish and cache is released cleanly.
- Load-test a synthetic burst and record how long the system stays above its latency objective.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Scale on queue wait | Tracks user-visible delay | Noisy under bursty traffic | Traffic is uniform and predictable |
| Scale on device utilisation | Simple metric, widely available | Saturation lags the queue | Long generations hold memory without compute |
| Aggressive scale-in | Lower idle cost | Discards warm cache and thrashes | Arrivals are bursty |
| High minimum replicas | Absorbs bursts immediately | Pays for idle capacity | Load is genuinely intermittent |
Good for
Tracks user-visible delay
Cost
Noisy under bursty traffic
Fails when
Traffic is uniform and predictable
- Traffic is uniform and predictable
- Long generations hold memory without compute
- Arrivals are bursty
- Load is genuinely intermittent
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Scale on queue wait is good for Tracks user-visible delay, costs Noisy under bursty traffic, and fails when Traffic is uniform and predictable. Scale on device utilisation is good for Simple metric, widely available, costs Saturation lags the queue, and fails when Long generations hold memory without compute. Aggressive scale-in is good for Lower idle cost, costs Discards warm cache and thrashes, and fails when Arrivals are bursty. High minimum replicas is good for Absorbs bursts immediately, costs Pays for idle capacity, and fails when Load is genuinely intermittent
Failure modes
- Flapping. Replicas are added and removed every few minutes and each new one pays cold start. Debug: plot replica count and cold-start count over an hour and look for alternation.
- Scale-out slower than the burst. The queue grows faster than replicas become ready and the objective is missed for the whole burst. Debug: compare burst arrival rate with replica readiness rate from the event log.
- Utilisation-triggered starvation. The autoscaler adds replicas on high utilisation, but the real limit is cache per replica. Debug: correlate queue wait with cache utilisation rather than compute utilisation.
In production
Alert on time above the latency objective, not on replica count. Keep a floor of warm capacity for the daily peak and review cold-start time whenever the model or image changes.
Common misconceptions
People think: Autoscaling removes the need for capacity planning.
Actually: It moves the problem to cold start and burst absorption; the floor and the ramp rate still have to be designed.
People think: Device utilisation is the right scaling signal.
Actually: Utilisation can be high while requests queue and low while memory is exhausted; queue-derived signals track user experience better.
Interview angle
Pick a leading signal, measure cold start, and explain the floor and the asymmetric scale-out and scale-in. Name the failure of scaling on utilisation. Saying it scales with load, with the reaction time unexamined, is the gap.
Sources
- Kubernetes, Horizontal Pod Autoscaling — metric-driven replica scaling and stabilisation windows.
- KEDA documentation — event-driven scaling on queue depth and custom metrics.
- Google, SRE Workbook: Monitoring — choosing signals that reflect user-visible behaviour.