Skip to content
AI Engineer Guide.
staffReportedcurated

A junior analyst gets an answer that quotes a privileged memo they are not staffed on. Firewalls and DLP saw nothing. What failed?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Do you compute ACLs at read time or at ingest, and what does each cost when permissions change?
  • Deriving ACLs at ingest makes the filter a cheap column read and lets the index pre-filter, but every permission change becomes an index write.
  • Computing at read time keeps the index simple and reflects a revocation immediately, at the cost of a lookup per query inside the search path.
  • The shape most teams ship is ingest-time ACLs stored as filterable metadata, refreshed by the same change feed as content, within a stated revocation target.
  • Test it by revoking access in the source, waiting one sync interval, then querying as that user and asserting zero hits.
How do you invalidate a semantic cache or a retrieved-set cache on revocation?
  • Key every cache entry on the full permission scope rather than the query text, so two identities asking the same thing cannot share an entry.
  • Keep a permission-set version per user, bump it on revocation, and make stale entries unreachable without scanning the cache.
  • A retrieved-set cache needs the same treatment as an answer cache, because it holds chunk IDs that outlive the grant that justified them.
  • Test with two synthetic users at different permission levels asking one question, and assert the narrow user never receives the broad answer.
What audit evidence proves the memo was never a candidate, not merely absent from the answer?
  • Log the permission scope, the eligible-document count and the candidate IDs with scores for each request, which shows what the search could reach.
  • Assert that every retrieval call carries a non-empty scope and fail closed when it does not, so a new route cannot silently widen access.
  • Run a scheduled negative test as the restricted user, turning the claim into a repeatable check rather than a one-off investigation.
  • Keep the record for the review window your policy requires, with chunk text redacted where PII rules demand it.

Red flags

  • prompt instruction
  • UI-only fix

Concepts this tests

Tags

Source: github.com

More in RAG & retrieval pipelines