staffReportedcurated
Latency, throughput and retrieval health all look normal, but users say answers are subtly out of date. Diagnose.
30-second answer
- Per-request metrics look healthy because they measure the wrong thing: the index is internally consistent, it is just old.
- Check index lag against the source-of-truth version per connector first, since a document-ID scheme change once left a team serving an eleven-day-old snapshot.
- Then compare sampled retrieved chunks against source updated_at and version, and alert on the staleness ratio rather than on job success.
- Real fixes are change-data-capture events instead of nightly rebuilds, explicitly propagated deletes, reconciliation jobs for dropped webhooks, and top-k overlap on a fixed probe set.
- Tighter freshness costs embedding spend and operational complexity, and an event stream buys eventual consistency with a measurable lag, not immediacy.
Likely follow-ups
Open a question to rehearse the answer.
How would you detect a connector that silently stopped syncing, before users notice?
- Alert on a watermark that has not advanced within its expected interval, per connector rather than in aggregate.
- Track index freshness per source and the age of the oldest vector, which catches a connector whose jobs still report success.
- Run a nightly probe set of questions whose answers change on known dates, the only reliable detector of a dead connector.
Should deletes and edits share one freshness SLO, and which is more dangerous when it slips?
- Deletes are more dangerous: a stale edit is merely outdated, while a missed delete keeps answering from a document that should no longer exist.
- Give deletes a tighter target and an explicit tombstone path, because a missing document is not a change event in most sources.
- Edits can absorb the source's update lag, while deletions cannot, since correctness and access both depend on them.
How do you measure index lag when the source system has no reliable updated_at?
- Hash document content and compare the source set against the index set, which detects change without trusting timestamps.
- Fall back to the connector's own cursor or change token and record it as the watermark even when it is opaque.
- Reconcile ID sets weekly and publish the measured lag rather than the lag you assumed.
Red flags
- scale replicas
- only check job success
Concepts this tests
Tags
Source: github.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?