Skip to content
AI Engineer Guide.

concept · concepts · beginner · concept 7 of 12

Choosing the right problem

0 of 12 complete

Why it matters

Teams often start with a model and search for a use, which produces demos that impress and products that do not ship. The work that determines success happens earlier: finding a decision with real cost, a tolerable error rate and a feedback signal. Those three properties decide whether iteration is possible at all, long before prompt or architecture choices matter.

Mental model

Problem selection is choosing which hill to climb, not how fast to climb it. The metaphor breaks because a hill has a height you can see, while the value of an AI feature is only revealed by shipping, so the choice is a bet with staged evidence.

How it works

Mechanism, step by stepInteractive
  1. 1Name the decision, who makes it today, how often, and what it costs when it is wrong.
  2. 2Check error tolerance: can the workflow absorb a wrong answer with review or a retry?
  3. 3Confirm a feedback signal exists: corrections, outcomes, or experts who can label.
  4. 4Estimate the value of a marginal improvement, not of perfection.
  5. 5Look for an existing baseline: a rule, a search box, a human process.
  6. 6Scope the first version to a slice where feedback is fastest and errors are cheapest.

Step 1 of 6

Name the decision, who makes it today, how often, and what it costs when it is wrong.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Name the decision, who makes it today, how often, and what it costs when it is wrong. 2. Check error tolerance: can the workflow absorb a wrong answer with review or a retry? 3. Confirm a feedback signal exists: corrections, outcomes, or experts who can label. 4. Estimate the value of a marginal improvement, not of perfection. 5. Look for an existing baseline: a rule, a search box, a human process. 6. Scope the first version to a slice where feedback is fastest and errors are cheapest.

  1. Name the decision, who makes it today, how often, and what it costs when it is wrong.
  2. Check error tolerance: can the workflow absorb a wrong answer with review or a retry?
  3. Confirm a feedback signal exists: corrections, outcomes, or experts who can label.
  4. Estimate the value of a marginal improvement, not of perfection.
  5. Look for an existing baseline: a rule, a search box, a human process.
  6. Scope the first version to a slice where feedback is fastest and errors are cheapest.

Trade-offs

ChoiceGood forCostFails when
High value, low toleranceLarge payoff if it worksLong evaluation and safety workNo feedback signal exists
Low value, high toleranceFast to ship and learnLittle business impactThe team needs a case for continued investment
Internal tooling firstCheap errors, fast feedbackLimited visibility and budgetThe product needs external users to generate data
Customer-facing firstDirect value and dataErrors are visible and costlyThe evaluation set cannot cover the input space
Trade-off mapInteractive

Good for

Large payoff if it works

Cost

Long evaluation and safety work

Fails when

No feedback signal exists

  1. No feedback signal exists
  2. The team needs a case for continued investment
  3. The product needs external users to generate data
  4. The evaluation set cannot cover the input space

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: High value, low tolerance is good for Large payoff if it works, costs Long evaluation and safety work, and fails when No feedback signal exists. Low value, high tolerance is good for Fast to ship and learn, costs Little business impact, and fails when The team needs a case for continued investment. Internal tooling first is good for Cheap errors, fast feedback, costs Limited visibility and budget, and fails when The product needs external users to generate data. Customer-facing first is good for Direct value and data, costs Errors are visible and costly, and fails when The evaluation set cannot cover the input space

Failure modes

  • Capability looking for a problem. A demo is built and no workflow adopts it. Debug: name the decision owner and the process change required; if either is missing, stop.
  • No feedback loop. Quality cannot improve because nobody records whether outputs were right. Debug: check that corrections are captured in a queryable store.
  • Intolerance discovered late. The workflow needs near-perfect accuracy and the model cannot provide it. Debug: measure the current human error rate and the review capacity before building.

In production

Instrument adoption and correction rate from the first release; both are evidence about problem fit. Set a review point where the problem is re-scoped if corrections do not fall.

Common misconceptions

People think: A good model makes a good product.

Actually: The same model fails or succeeds depending on error tolerance and feedback, which are properties of the problem.

People think: Start with the hardest problem to prove capability.

Actually: Hard problems rarely give fast feedback, so the team learns slowly and cannot tell whether the approach works.

Interview angle

Strong answers begin with a decision, its owner and the cost of an error, then check error tolerance and the feedback signal before discussing models. Weak answers open with a technique and work backwards to a use case.

Sources