Skip to content
AI Engineer Guide.
staffReportedcurated

Latency, throughput and retrieval health all look normal, but users say answers are subtly out of date. Diagnose.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How would you detect a connector that silently stopped syncing, before users notice?
  • Alert on a watermark that has not advanced within its expected interval, per connector rather than in aggregate.
  • Track index freshness per source and the age of the oldest vector, which catches a connector whose jobs still report success.
  • Run a nightly probe set of questions whose answers change on known dates, the only reliable detector of a dead connector.
Should deletes and edits share one freshness SLO, and which is more dangerous when it slips?
  • Deletes are more dangerous: a stale edit is merely outdated, while a missed delete keeps answering from a document that should no longer exist.
  • Give deletes a tighter target and an explicit tombstone path, because a missing document is not a change event in most sources.
  • Edits can absorb the source's update lag, while deletions cannot, since correctness and access both depend on them.
How do you measure index lag when the source system has no reliable updated_at?
  • Hash document content and compare the source set against the index set, which detects change without trusting timestamps.
  • Fall back to the connector's own cursor or change token and record it as the watermark even when it is opaque.
  • Reconcile ID sets weekly and publish the measured lag rather than the lag you assumed.

Red flags

  • scale replicas
  • only check job success

Concepts this tests

Tags

Source: github.com

More in RAG & retrieval pipelines