Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 17 of 24

Rate limiting

0 of 24 complete

Why it matters

Every shared resource has a capacity, and without a limit the first caller to arrive with a loop consumes it. Rate limiting makes the allocation explicit: how much each caller may use, over what window, and what happens when they exceed it. The design questions are the algorithm, because it determines burst tolerance; the state, because it determines whether the limit is global; and the response, because a client that cannot tell when to retry will retry immediately.

Mental model

It is a turnstile that counts entries per minute: it protects the room, and the queue outside is visible. The analogy breaks because the turnstile has one counter, while a distributed limiter has to agree on the count across several doors.

How it works

Mechanism, step by stepInteractive
  1. 1Choose the dimension from the resource being protected: requests per minute for compute, tokens per minute for model quota, concurrent jobs for memory.
  2. 2Pick the algorithm for the burst behaviour you want, and document how a client that sends a burst is treated.
  3. 3Keep counters in a shared store with atomic increments and a lifetime, or accept that limits are per instance.
  4. 4Return a clear status with the limit, the remaining allowance and a retry hint.
  5. 5Apply limits per tenant and per endpoint, and keep a separate bucket for internal or admin traffic.
  6. 6Load-test the limiter itself, because it sits on every request and its store can become the bottleneck.

Step 1 of 6

Choose the dimension from the resource being protected: requests per minute for compute, tokens per minute for model quota, concurrent jobs for memory.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Choose the dimension from the resource being protected: requests per minute for compute, … 2. Pick the algorithm for the burst behaviour you want, and document how a client that sends… 3. Keep counters in a shared store with atomic increments and a lifetime, or accept that lim… 4. Return a clear status with the limit, the remaining allowance and a retry hint. 5. Apply limits per tenant and per endpoint, and keep a separate bucket for internal or admi… 6. Load-test the limiter itself, because it sits on every request and its store can become t…

  1. Choose the dimension from the resource being protected: requests per minute for compute, tokens per minute for model quota, concurrent jobs for memory.
  2. Pick the algorithm for the burst behaviour you want, and document how a client that sends a burst is treated.
  3. Keep counters in a shared store with atomic increments and a lifetime, or accept that limits are per instance.
  4. Return a clear status with the limit, the remaining allowance and a retry hint.
  5. Apply limits per tenant and per endpoint, and keep a separate bucket for internal or admin traffic.
  6. Load-test the limiter itself, because it sits on every request and its store can become the bottleneck.

Trade-offs

ChoiceGood forCostFails when
Token bucketAllows controlled burstsA burst can still overwhelm a fragile backendThe backend cannot absorb any burst
Fixed windowSimple and cheapDouble the limit at the boundaryTraffic is bursty near window edges
Sliding windowSmoother enforcementMore state and computationThe limiter must be very cheap
Per-instance countersNo shared dependencyEffective limit scales with replicasThe limit is a contractual number
Trade-off mapInteractive

Good for

Allows controlled bursts

Cost

A burst can still overwhelm a fragile backend

Fails when

The backend cannot absorb any burst

  1. The backend cannot absorb any burst
  2. Traffic is bursty near window edges
  3. The limiter must be very cheap
  4. The limit is a contractual number

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Token bucket is good for Allows controlled bursts, costs A burst can still overwhelm a fragile backend, and fails when The backend cannot absorb any burst. Fixed window is good for Simple and cheap, costs Double the limit at the boundary, and fails when Traffic is bursty near window edges. Sliding window is good for Smoother enforcement, costs More state and computation, and fails when The limiter must be very cheap. Per-instance counters is good for No shared dependency, costs Effective limit scales with replicas, and fails when The limit is a contractual number

Failure modes

  • Split-brain counters. Replicas each allow the full limit and the aggregate is a multiple of the intended rate. Debug: compare total admitted requests with the configured limit while scaling replicas.
  • Boundary burst. A client sends the full allowance at the end of one window and the start of the next. Debug: chart admitted requests in short intervals around the window boundary.
  • Limiter as bottleneck. The shared counter store adds latency to every request and becomes the outage. Debug: measure limiter latency separately and check its error rate during peak.

In production

Monitor rejection rate by tenant, limiter latency and store availability. Keep a documented fallback behaviour for when the limiter store is unreachable, and make sure it fails in the direction you chose.

Common misconceptions

People think: Rate limiting is the same as throttling.

Actually: A limit rejects or delays at admission; throttling can also slow or degrade work after it is admitted.

People think: A per-instance limit is good enough.

Actually: It is only good enough if you accept that the effective limit grows with the number of instances.

Interview angle

Pick the dimension, the algorithm and the shared state, then the response the client receives. Name the limiter's own failure mode. A counter in memory, with no thought about a second replica, is incomplete.

Sources