Why it matters
Every shared resource has a capacity, and without a limit the first caller to arrive with a loop consumes it. Rate limiting makes the allocation explicit: how much each caller may use, over what window, and what happens when they exceed it. The design questions are the algorithm, because it determines burst tolerance; the state, because it determines whether the limit is global; and the response, because a client that cannot tell when to retry will retry immediately.
Mental model
It is a turnstile that counts entries per minute: it protects the room, and the queue outside is visible. The analogy breaks because the turnstile has one counter, while a distributed limiter has to agree on the count across several doors.
How it works
- 1Choose the dimension from the resource being protected: requests per minute for compute, tokens per minute for model quota, concurrent jobs for memory.
- 2Pick the algorithm for the burst behaviour you want, and document how a client that sends a burst is treated.
- 3Keep counters in a shared store with atomic increments and a lifetime, or accept that limits are per instance.
- 4Return a clear status with the limit, the remaining allowance and a retry hint.
- 5Apply limits per tenant and per endpoint, and keep a separate bucket for internal or admin traffic.
- 6Load-test the limiter itself, because it sits on every request and its store can become the bottleneck.
Step 1 of 6
Choose the dimension from the resource being protected: requests per minute for compute, tokens per minute for model quota, concurrent jobs for memory.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Choose the dimension from the resource being protected: requests per minute for compute, … 2. Pick the algorithm for the burst behaviour you want, and document how a client that sends… 3. Keep counters in a shared store with atomic increments and a lifetime, or accept that lim… 4. Return a clear status with the limit, the remaining allowance and a retry hint. 5. Apply limits per tenant and per endpoint, and keep a separate bucket for internal or admi… 6. Load-test the limiter itself, because it sits on every request and its store can become t…
- Choose the dimension from the resource being protected: requests per minute for compute, tokens per minute for model quota, concurrent jobs for memory.
- Pick the algorithm for the burst behaviour you want, and document how a client that sends a burst is treated.
- Keep counters in a shared store with atomic increments and a lifetime, or accept that limits are per instance.
- Return a clear status with the limit, the remaining allowance and a retry hint.
- Apply limits per tenant and per endpoint, and keep a separate bucket for internal or admin traffic.
- Load-test the limiter itself, because it sits on every request and its store can become the bottleneck.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Token bucket | Allows controlled bursts | A burst can still overwhelm a fragile backend | The backend cannot absorb any burst |
| Fixed window | Simple and cheap | Double the limit at the boundary | Traffic is bursty near window edges |
| Sliding window | Smoother enforcement | More state and computation | The limiter must be very cheap |
| Per-instance counters | No shared dependency | Effective limit scales with replicas | The limit is a contractual number |
Good for
Allows controlled bursts
Cost
A burst can still overwhelm a fragile backend
Fails when
The backend cannot absorb any burst
- The backend cannot absorb any burst
- Traffic is bursty near window edges
- The limiter must be very cheap
- The limit is a contractual number
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Token bucket is good for Allows controlled bursts, costs A burst can still overwhelm a fragile backend, and fails when The backend cannot absorb any burst. Fixed window is good for Simple and cheap, costs Double the limit at the boundary, and fails when Traffic is bursty near window edges. Sliding window is good for Smoother enforcement, costs More state and computation, and fails when The limiter must be very cheap. Per-instance counters is good for No shared dependency, costs Effective limit scales with replicas, and fails when The limit is a contractual number
Failure modes
- Split-brain counters. Replicas each allow the full limit and the aggregate is a multiple of the intended rate. Debug: compare total admitted requests with the configured limit while scaling replicas.
- Boundary burst. A client sends the full allowance at the end of one window and the start of the next. Debug: chart admitted requests in short intervals around the window boundary.
- Limiter as bottleneck. The shared counter store adds latency to every request and becomes the outage. Debug: measure limiter latency separately and check its error rate during peak.
In production
Monitor rejection rate by tenant, limiter latency and store availability. Keep a documented fallback behaviour for when the limiter store is unreachable, and make sure it fails in the direction you chose.
Common misconceptions
People think: Rate limiting is the same as throttling.
Actually: A limit rejects or delays at admission; throttling can also slow or degrade work after it is admitted.
People think: A per-instance limit is good enough.
Actually: It is only good enough if you accept that the effective limit grows with the number of instances.
Interview angle
Pick the dimension, the algorithm and the shared state, then the response the client receives. Name the limiter's own failure mode. A counter in memory, with no thought about a second replica, is incomplete.
Sources
- RFC 6585, Additional HTTP Status Codes — the throttling status and Retry-After semantics.
- IETF, RateLimit header fields for HTTP — headers that communicate quota and reset.
- Stripe, Rate limits — a production limiter with per-endpoint behaviour and read limits.