Skip to content
AI Engineer Guide.
seniorReportedcurated

Tell me about a model-quality regression you introduced or discovered after deployment. What did you do?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Which guardrail missed it, and why?
  • Aggregate scoring missed it, because a slice falling from 86% to 61% was small enough to disappear inside the average.
  • The eval set had no identifier queries, so the failure mode was outside what the suite was looking at.
  • Staging traffic did not carry the error-code distribution, which is why a canary on real traffic should precede a full index swap.
How would you have caught it before any user saw it?
  • Build the shadow index, replay the labelled set against both indexes, and compare recall split by query type rather than overall.
  • Canary the new index on 10% of traffic with an alert on the identifier slice, so the detection window is a subset of users rather than all of them.
  • Add a probe set of known error codes that must return their exact page, because that check is cheap and fails loudly.

Red flags

  • Attributes the regression to model nondeterminism with no containment story
  • Ships a fix without adding a permanent test for the failing slice

Concepts this tests

Tags

Source: www.onetworesume.com

More in Behavioural & project deep-dives