Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 31 of 38

Rate-limit budgeting

0 of 38 complete

Why it matters

Provider quotas are shared by every feature and tenant that uses the same key, and they are consumed by work you have not sent yet: a request that will produce a long answer needs headroom before it starts. Without a budget, the first symptom is a rejection during a launch, followed by retries that consume the remaining quota. Reserving capacity per unit makes the limit an input to scheduling rather than an error to handle.

Mental model

It is a table reservation: the seat is held before you arrive, so the restaurant does not overbook and turn people away at the door. The analogy breaks because a reservation can be cancelled and reused, while an unused token allowance in a minute window usually cannot be banked.

How it works

Two limits, one boundaryInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Requests served and requests refused against offered load, under a 400,000 token-per-minute allowance and a 600 request-per-minute ceiling. The quota is illustrative; the boundary is wherever the binding limit falls.

What to notice. With 1,000-token requests the token allowance admits 400 requests a minute, below the 600 ceiling, so everything past 400 is a 429. With 4,000-token requests the same allowance admits 100: the boundary moves four times further left while the request ceiling never moves. Prompt size, not traffic volume, decides which limit binds.

Text version

Offered against served requests per minute from 0 to 800. The served area rises with offered load to 400 and then flattens; a dashed refused line starts at 400 and reaches 400 at 800 offered. A vertical 429 boundary line sits at 400 requests a minute and a request ceiling line at 600. Two buttons switch the request size to 4,000 tokens: the boundary then moves to 100 requests a minute and the refused line covers most of the range, against the unchanged 600 ceiling.

  1. Record the provider's limits per key, per model and per region, and note which limit applies to input plus output.
  2. Estimate tokens for each request from the rendered prompt plus the maximum output you will allow.
  3. Reserve from a shared bucket before sending, and release the difference when actual usage is known.
  4. Partition buckets by tenant and feature so a batch job cannot starve interactive traffic.
  5. Queue with a deadline when the bucket is empty, and reject early when the wait would exceed it.
  6. Honour retry hints and apply jitter, and keep concurrency limits separate from rate limits.

Trade-offs

ChoiceGood forCostFails when
Global bucketSimple and fully utilisedOne tenant can starve the restMultiple tenants share a key
Per-tenant bucketsFairness and isolationUnused allowance per tenantTenants have very different sizes
Estimate from maximum outputSafe reservationsUnder-utilisation when answers are shortOutput length is predictable
Estimate from expected outputHigher utilisationOvershoot and throttlingLong answers are common
Trade-off mapInteractive

Good for

Simple and fully utilised

Cost

One tenant can starve the rest

Fails when

Multiple tenants share a key

  1. Multiple tenants share a key
  2. Tenants have very different sizes
  3. Output length is predictable
  4. Long answers are common

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Global bucket is good for Simple and fully utilised, costs One tenant can starve the rest, and fails when Multiple tenants share a key. Per-tenant buckets is good for Fairness and isolation, costs Unused allowance per tenant, and fails when Tenants have very different sizes. Estimate from maximum output is good for Safe reservations, costs Under-utilisation when answers are short, and fails when Output length is predictable. Estimate from expected output is good for Higher utilisation, costs Overshoot and throttling, and fails when Long answers are common

Failure modes

  • Reservation ignores output. Input-only estimates pass the check and the response pushes usage past the limit. Debug: reconcile reserved tokens against billed tokens per request and look for systematic undershoot.
  • Retry storm on rejection. Clients retry immediately and consume the remaining allowance in failed calls. Debug: chart throttled responses against request rate and check that backoff is actually applied.
  • Shared key across features. An embedding backfill exhausts the generation quota because both use one key. Debug: split usage metrics by feature and compare with the provider's per-key report.

In production

Monitor quota headroom, throttling rate and queue wait per bucket. Keep a reserve for interactive traffic, and alert when any tenant consumes a large share of the shared allowance.

Common misconceptions

People think: Rate limits only matter at high traffic.

Actually: A single long-context request can consume a large share of a token-per-minute allowance, so limits bind on prompt size as much as on volume.

People think: Retrying a throttled request is harmless.

Actually: Retries consume quota and can extend the throttling window, which is why the retry hint and jitter exist.

Interview angle

Separate request and token limits, reserve from an estimate that includes output, and partition buckets by tenant. Explain what happens when the bucket is empty. A retry loop presented as the whole throttling strategy is insufficient.

Sources