seniorReportedcurated
Tell me about a model-quality regression you introduced or discovered after deployment. What did you do?
30-second answer
- I found it at the slice, not the average: the aggregate answer score barely moved while citation precision dropped after an embedding upgrade.
- I contained it before diagnosing, rolling the index back the same afternoon because a live quality regression is an incident and the old index is a known-good state.
- The cause was that the new model over-clustered short alphanumeric error codes, because a lexical signal had been dropped from the retrieval path.
- Restoring BM25 fusion added index complexity and a tuning surface, but it recovered the failing slice, and the hybrid configuration improved grounded accuracy by 7 points without repeating the regression.
- I made the slice permanent: 120 error-code queries went into the eval suite, and embedding changes now go to 10% of traffic first, because one aggregate metric hid a slice falling from 86% to 61%.
- The rollback was cheap only because the index was rebuildable, so a migration that cannot be reversed needs a shadow index and a slice-level comparison before cutover.
Likely follow-ups
Open a question to rehearse the answer.
Which guardrail missed it, and why?
- Aggregate scoring missed it, because a slice falling from 86% to 61% was small enough to disappear inside the average.
- The eval set had no identifier queries, so the failure mode was outside what the suite was looking at.
- Staging traffic did not carry the error-code distribution, which is why a canary on real traffic should precede a full index swap.
How would you have caught it before any user saw it?
- Build the shadow index, replay the labelled set against both indexes, and compare recall split by query type rather than overall.
- Canary the new index on 10% of traffic with an alert on the identifier slice, so the detection window is a subset of users rather than all of them.
- Add a probe set of known error codes that must return their exact page, because that check is cheap and fails loudly.
Red flags
- Attributes the regression to model nondeterminism with no containment story
- Ships a fix without adding a permanent test for the failing slice
Concepts this tests
Tags
Source: www.onetworesume.com
More in Behavioural & project deep-dives
- Walk me through an AI feature you shipped end to end and your exact role. What decisions were yours?
- Tell me about a time a model surprised you in production. How did you find out, and what did you change?
- Describe a time you disagreed with a product or engineering stakeholder about using an agent, fine-tuning, or RAG.