Why it matters
Safety rules change faster than product code and are often owned by a different team. Embedding them in the application means every policy change is an application deploy, and every service reimplements the same checks. A sidecar centralises them, at the cost of another hop and a shared dependency in the request path, which is a real availability decision rather than a detail.
Mental model
A security checkpoint at a building entrance: everyone passes through and policy is set centrally. The metaphor breaks because a checkpoint can be walked around if another door exists, and a sidecar is bypassable whenever a client can reach the model directly.
How it works
Step 1 of 5
The client reaches only the app
The client holds no provider credential, so the application is the entrance. Removing that credential is a deployment decision, not a policy setting.
Active layers
- Client
- Application
Hover a layer to read what it holds.
Next: Enter the sidecar
- Position
- one hop, both ways
- Fail mode
- chosen per check
- Bypass
- direct provider call
Figure. Every request the application serves passes through the sidecar: input check, provider call, output check, and a check on tool arguments before execution. A text answer goes straight from the output check to the response. The dashed lane runs from the batch-job band to the unchecked provider band: that caller has its own key, so no check applies. Timeouts, retries and the application's own telemetry are omitted.
What to notice. The sidecar holds the credential the application uses, so its client has no route to the model except through this stack. Nothing in the sidecar inspects a call made around it: the batch job's dashed edge skips every check and still reaches the provider, which is why the two provider bands are drawn separately. Comparing provider request counts with sidecar request counts is what makes that path visible.
Text version
A vertical stack. A client holding no provider key calls the application, which routes through a sidecar group. The group holds four bands: input check, the provider call that holds the key, an output check, and a tool-argument check. The output check has two exits: a direct edge labelled text to the response, and the tool path through the argument check and tool execution. Below the response sit the batch job and the unchecked provider. A dashed lane runs from the batch job to that unchecked provider, labelled holds a key, skipping every check in the sidecar. Five steps cover the client entrance, entering the sidecar, the provider call, the output and tool checks, and the cost of a bypass.
- List the checks and their position: input classification before the model, output filtering after, tool-argument checks before execution.
- Decide fail-open or fail-closed per check, and write down the consequence of each during an outage.
- Set a latency budget for the sidecar and measure it separately from the model call.
- Version policy independently and log the policy version with every decision.
- Make the application's client the only route to the model, so the sidecar cannot be bypassed.
- Test with adversarial inputs in continuous integration, including encoded and multi-turn attempts.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Sidecar proxy | Language-agnostic, central policy | Extra hop and a new failure point | The latency budget is tight |
| In-process library | No network hop | Policy coupled to each service's deploy | Many services need one policy |
| Fail-closed | Strong guarantee | An outage becomes a product outage | Availability is the priority |
| Fail-open | The product stays up | Policy silently off during incidents | The risk is regulated |
Good for
Language-agnostic, central policy
Cost
Extra hop and a new failure point
Fails when
The latency budget is tight
- The latency budget is tight
- Many services need one policy
- Availability is the priority
- The risk is regulated
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Sidecar proxy is good for Language-agnostic, central policy, costs Extra hop and a new failure point, and fails when The latency budget is tight. In-process library is good for No network hop, costs Policy coupled to each service's deploy, and fails when Many services need one policy. Fail-closed is good for Strong guarantee, costs An outage becomes a product outage, and fails when Availability is the priority. Fail-open is good for The product stays up, costs Policy silently off during incidents, and fails when The risk is regulated
Failure modes
- Bypass path. A batch job or internal tool calls the provider directly. Debug: compare provider request counts with sidecar request counts over a day.
- Fail-open unnoticed. The sidecar is down for hours and nothing alerts. Debug: alert on sidecar health and on the ratio of checked to unchecked requests.
- Policy drift. Two sidecar versions run during a rollout and decisions differ. Debug: log the policy version per decision and assert one version per fleet.
In production
Monitor sidecar latency, error rate and policy version coverage. Rehearse an outage and confirm the chosen fail mode behaves as documented.
Common misconceptions
People think: A guardrail makes the system safe.
Actually: It enforces the policies you wrote at the points you covered; uncovered paths and novel inputs remain.
People think: Sidecars are free because they are small.
Actually: Every request pays a hop, and the sidecar becomes a dependency with its own availability.
Interview angle
Place the checks, choose fail modes and name the bypass risk. Strong answers discuss policy versioning and latency. Weak ones add a filter at the end of the pipeline and call the system guarded.
Sources
- Azure Architecture Center, Sidecar Pattern — co-deployed components and their lifecycle.
- Rebedea et al., NeMo Guardrails — programmable rails for controlling model interactions.
- OWASP Top 10 for Large Language Model Applications — the risks such a layer is meant to cover.