Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 3 of 3

Observability tools compared

0 of 3 complete

Why it matters

A model call is opaque by default: the prompt, the retrieved context, the sampled output and the latency all live inside one HTTP request. Without traces you cannot answer why a specific answer was wrong, and without cost attribution you cannot tell which feature is expensive. Observability tooling exists to make those questions answerable after the fact, but each tool captures a different slice of the request.

Mental model

Observability is a flight recorder, not an autopilot. It tells you what happened on the flight you already took. The metaphor breaks because LLM traces are also training and evaluation data, so retention and access rules matter more than they do for logs.

How it works

One request, three capture points, one export lagInteractive
workwaitriskhidden from the SDKevent
0.06 s

Event · step 1 of 7

0.06 s · the gateway routes

At 0.06 s

The request enters a gateway that holds the provider credentials and the retry policy. The app SDK opens one span around the call; the gateway opens its own span per attempt.

On screen at this event

  • route
  • call
  • 429

Next at 1.2 s: 1.2 s · attempt one is rate-limited

Attempts done
0
SDK spans
0
Trace in store
no
Flush lag
—
Queryable after
—
Step 1/7 · 0.06 s · the gateway routes

Figure. A synthetic 2.5 s request: the gateway sees a 429 and a retry, the in-path SDK sees one 2.34 s call, and the collector begins exporting 0.9 s after the answer, with the span queryable one second after. A deployment usually runs one or two of these points; all three are drawn on one request to compare coverage. Times are illustrative.

What to notice. The capture point decides the evidence, not the vendor. The SDK has the prompt and the tool arguments but not the gateway retry; the gateway has the two attempts and no application context. Batch export keeps the backend cheap and leaves the freshest incident unqueryable. Sampling, redaction and per-field retention are not shown, and they decide whether the evidence exists at the review.

Text version

Six lanes over 4.2 seconds: client, gateway, app SDK, provider, collector and trace store. The client waits 0 to 2.5 and reads 2.5 to 3.7. The gateway routes at 0.02 to 0.06, attempt 1 0.06 to 1.2 in red, a 60 ms wait, attempt 2 1.26 to 2.4 and a response 2.4 to 2.5. The provider returns 429 at 0.08 to 1.2 and 200 at 1.28 to 2.4. The SDK holds one span 0.06 to 2.4 with a dashed not seen 60 ms interval at the retry. The collector waits 0.06 to 3.4 and exports 3.4 to 3.5; the store is queryable 3.5 to 4.2. Markers read rate-limited at 1.2 and indexed at 3.5. Counters: attempts done 0, 1, 1, 2, 2, 2, 2; SDK spans one from 2.4; trace in store no until 3.5; flush lag 0.9 s and queryable 1.0 s from the 3.5 s step.

  1. List the questions you need to answer in an incident: which prompt version, which retrieved chunks, which model, what latency, what cost.
  2. Choose the capture point: an in-process SDK, a gateway proxy, or OpenTelemetry spans emitted by your own code.
  3. Define the trace schema. request id, conversation id, model, prompt version, token counts, tool calls, retrieval ids — before choosing a vendor.
  4. Decide redaction and retention per field; prompts and tool outputs frequently contain personal data.
  5. Wire sampling: full capture for errors and a small deterministic sample for successes.
  6. Connect traces to evaluation runs so a failing slice can be replayed against a candidate.

Trade-offs

ChoiceGood forCostFails when
In-path SDKFast to adopt, rich contextVendor lock-in, latency and failure surface in the request pathYou need strict isolation or multi-vendor portability
Gateway proxyLanguage-agnostic, central policyAnother hop, limited application contextYou need business-level attributes
OpenTelemetry onlyPortable, standard toolingYou build the LLM-specific viewsThe team has no observability platform
Full prompt captureBest debuggingStorage cost and privacy exposurePrompts contain regulated data
Trade-off mapInteractive

Good for

Fast to adopt, rich context

Cost

Vendor lock-in, latency and failure surface in the request path

Fails when

You need strict isolation or multi-vendor portability

  1. You need strict isolation or multi-vendor portability
  2. You need business-level attributes
  3. The team has no observability platform
  4. Prompts contain regulated data

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: In-path SDK is good for Fast to adopt, rich context, costs Vendor lock-in, latency and failure surface in the request path, and fails when You need strict isolation or multi-vendor portability. Gateway proxy is good for Language-agnostic, central policy, costs Another hop, limited application context, and fails when You need business-level attributes. OpenTelemetry only is good for Portable, standard tooling, costs You build the LLM-specific views, and fails when The team has no observability platform. Full prompt capture is good for Best debugging, costs Storage cost and privacy exposure, and fails when Prompts contain regulated data

Failure modes

  • Trace without a join key. You cannot connect an answer to the retrieval call that produced it. Debug: propagate one request id through every span and assert it is present.
  • Unsampled success data. Dashboards describe errors only and miss slow quality drift. Debug: sample successes deterministically by request id.
  • Retention by default. Prompt logs outlive the purpose they were collected for. Debug: set per-field TTLs and verify deletion with a test.

In production

Instrument once, at the boundary where you already control the request: a gateway or a shared client. Emit OpenTelemetry spans so the data survives a vendor change, and keep an export path to object storage for offline evaluation.

Common misconceptions

People think: Observability and evaluation are the same purchase.

Actually: Tracing records what happened; evaluation judges whether it was good. A tracer with no labelled set cannot tell you either.

People think: Capturing every prompt is necessary for debugging.

Actually: A sampled, redacted trace plus full capture on errors answers most incident questions at a fraction of the risk.

Interview angle

Discuss the questions you need to answer, then the capture point and its failure modes. Mention redaction, retention and the join key between retrieval and generation. A weak answer lists vendors; a strong one explains what data model makes a regression debuggable.

Sources