staffReportedcurated
How do you decide when a model or agent is good enough to ship?
30-second answer
- I reframed good enough per slice rather than in aggregate, because a wrong summary on a standard clause is cheap and a missed exception is not, and the average hides that asymmetry.
- The bar was a conjunction: cited evidence for every claim, an abstain path when evidence is missing, and human escalation for nonstandard agreements.
- That bar had a visible cost, since abstention cut coverage by roughly 12% of queries and looked worse in demos, which is the trade I would defend.
- I would ship when the failing slice is bounded and escalated, not when the average passes, and the first quarter supported that with 68% of routine clause questions resolved without attorney triage and zero unsupported approvals.
- The decision flips if the abstain rate rises without a matching fall in errors, because that is a retrieval problem wearing a safety costume.
- A single accuracy number, in this case 91% on summaries, is a demo metric rather than a ship criterion, because it cannot tell you which failures are tolerable.
Likely follow-ups
Open a question to rehearse the answer.
What would have made you delay the launch?
- I would delay the launch for any unsupported approval on the standard slice, because that is the failure the product cannot absorb, regardless of the aggregate score.
- I would also delay if the abstain path failed on known-unanswerable questions, or if citation verification passed claims absent from the cited text.
- A queue that could not absorb the routed volume would delay it too, since a gate nobody staffs is not a control.
Who owns the decision when product and legal disagree on the bar?
- The accountable risk owner sets the failure tolerance, because legal owns the consequence of a missed exception and product owns coverage.
- Engineering supplies the measurement and states plainly when the evidence does not support the bar either side wants.
- The decision and its threshold should be written down before launch, so a later disagreement is about new evidence rather than a renegotiation.
Red flags
- Gives a single accuracy number as the ship criterion
- Optimises for demo quality rather than downstream consequence
Concepts this tests
Tags
Source: www.onetworesume.com
More in Behavioural & project deep-dives
- Walk me through an AI feature you shipped end to end and your exact role. What decisions were yours?
- Tell me about a time a model surprised you in production. How did you find out, and what did you change?
- Describe a time you disagreed with a product or engineering stakeholder about using an agent, fine-tuning, or RAG.