Skip to content
AI Engineer Guide.
staffReportedcurated

How do you decide when a model or agent is good enough to ship?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What would have made you delay the launch?
  • I would delay the launch for any unsupported approval on the standard slice, because that is the failure the product cannot absorb, regardless of the aggregate score.
  • I would also delay if the abstain path failed on known-unanswerable questions, or if citation verification passed claims absent from the cited text.
  • A queue that could not absorb the routed volume would delay it too, since a gate nobody staffs is not a control.
Who owns the decision when product and legal disagree on the bar?
  • The accountable risk owner sets the failure tolerance, because legal owns the consequence of a missed exception and product owns coverage.
  • Engineering supplies the measurement and states plainly when the evidence does not support the bar either side wants.
  • The decision and its threshold should be written down before launch, so a later disagreement is about new evidence rather than a renegotiation.

Red flags

  • Gives a single accuracy number as the ship criterion
  • Optimises for demo quality rather than downstream consequence

Concepts this tests

Tags

Source: www.onetworesume.com

More in Behavioural & project deep-dives