Why it matters
Without a shared layer, every service holds its own provider key, implements its own retries and reports usage in its own format, so quotas are invisible until they are exhausted and cost cannot be attributed. A gateway concentrates those concerns: one place to rotate keys, enforce per-tenant limits, emit traces and switch providers. The cost is a new single point of failure on the request path, which is why streaming behaviour and failure mode matter more than the feature list.
Mental model
It is a building's reception desk: visitors are checked, routed and logged in one place. The analogy breaks when the receptionist is also the only person who can open the door, so a queue at the desk stops the whole building.
How it works
The fleet is replaceable; the shared counters and the bypass path are what the design actually depends on.
Step 1 of 5
Reach the fleet
Every caller reaches one stateless name. The fleet is a shared failure domain, so it needs the same availability target, capacity plan and on-call as the application it serves.
service
Gateway fleet
Gateway fleet
3 replicas · stateless
Several replicas behind one name, with no per-replica session state. Capacity must cover losing one replica, because the fleet is on every request path including health checks. Three is an example, not a threshold.
12 connections
Figure. The topology places one stateless gateway fleet in the path of the managed calls, with quota reservations and credentials in shared stores, so replicas can be replaced without losing state. It does not guarantee the fleet is not a single point of failure, that failover capacity exists, or that every caller uses it: those are capacity, game-day and coverage questions.
What to notice. The fleet is drawn as three stacked replicas because statelessness is the design, not a detail; three is an example, not a threshold. The only request-path state is the shared quota counter. The red edges are a provider limit that can trigger failover, a fleet that returns failures to an application, and a caller outside the gateway that is invisible to quota and traces.
Text version
A production LLM gateway topology. Two interactive applications and a batch job call a stateless gateway fleet drawn as three replicas. Any replica reserves quota in a shared atomic counter, mints a short-lived provider token from a key store, and streams through the primary provider. Traces and usage are written separately from provider latency. A classified provider failure can fail over to a backup with reserved capacity. Red dashed failure edges show the batch job calling a provider directly with quota blind, and the gateway fleet returning a 503 to an application. Five steps cover reaching the fleet, reserving quota and minting tokens, streaming through, absorbing a provider failure, and finding partial coverage.
- Put the gateway in the path of every model call, including batch jobs and internal tools, or the quota picture stays partial.
- Terminate provider credentials at the gateway and issue per-application tokens with scopes.
- Normalise requests and responses into one schema, and keep provider-specific options in an extension field.
- Stream responses through without buffering the body, and propagate client disconnects upstream.
- Enforce per-tenant quotas, caching and fallback policy at this layer, and emit one trace per call.
- Deploy the gateway with the same canary and rollback discipline as the model, since a bad config is an outage.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Self-hosted gateway | Full control and data stays inside | You operate a critical service | The team is small and cannot staff it |
| Managed gateway | Fast adoption and updates | Another vendor in the data path | Prompts contain regulated data |
| Gateway-level caching | Reuse across applications | Shared key design and leakage risk | Tenants have different permissions |
| Thin pass-through | Low latency, few assumptions | Little central policy | Every team reimplements quotas |
Good for
Full control and data stays inside
Cost
You operate a critical service
Fails when
The team is small and cannot staff it
- The team is small and cannot staff it
- Prompts contain regulated data
- Tenants have different permissions
- Every team reimplements quotas
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Self-hosted gateway is good for Full control and data stays inside, costs You operate a critical service, and fails when The team is small and cannot staff it. Managed gateway is good for Fast adoption and updates, costs Another vendor in the data path, and fails when Prompts contain regulated data. Gateway-level caching is good for Reuse across applications, costs Shared key design and leakage risk, and fails when Tenants have different permissions. Thin pass-through is good for Low latency, few assumptions, costs Little central policy, and fails when Every team reimplements quotas
Failure modes
- Buffered streaming. Tokens arrive only when the response completes and perceived latency collapses. Debug: measure time to first byte at the gateway and compare with the provider's direct response.
- Gateway as a single point of failure. Every application fails when the gateway is unhealthy, including the health checks. Debug: run a game day that kills the gateway and measure application behaviour.
- Partial coverage. A batch job calls the provider directly and bypasses quotas, so limits are hit unexpectedly. Debug: compare provider-side request counts with gateway-side counts for the same day.
In production
Track gateway latency separately from provider latency, and alert on added overhead and stream stalls. Keep a documented bypass path for emergencies, with the quota and tracing consequences written down.
Common misconceptions
People think: A gateway is just a reverse proxy.
Actually: It carries policy: credentials, quotas, routing, caching and trace semantics that applications would otherwise duplicate.
People think: Adding a gateway always improves control.
Actually: It centralises control and concentrates risk; the request path now has one more component that must stream, scale and fail well.
Interview angle
List what the gateway centralises, then the risk it introduces and how to bound it. Mention streaming passthrough and partial coverage. Treating it as a proxy with no policy, or as free abstraction, misses both sides.
Sources
- LiteLLM documentation — a gateway that normalises provider APIs, budgets and routing.
- Envoy Proxy documentation — proxy behaviour for streaming, retries and load balancing.