Skip to content
AI Engineer Guide.

concept · concepts · beginner · concept 11 of 12

AI quality attributes

0 of 12 complete

Why it matters

A model that answers correctly in a notebook can still be unusable: too slow for the interface, too expensive per request, wrong for a subset of users, or impossible to debug. Quality attributes make those constraints explicit and measurable, so design decisions can be argued with evidence rather than anecdotes, and the loudest demo stops winning the review.

Mental model

Quality attributes are the load limits on a bridge, not the bridge's length. The metaphor breaks because load limits are fixed by physics and certified, while these attributes are estimates that shift with data and must be re-measured as traffic changes.

How it works

Mechanism, step by stepInteractive
  1. 1List the attributes that matter for this product and the decision each one constrains.
  2. 2Give each a metric, a target and a tolerance: p95 latency, cost per request, refusal rate on a labelled set.
  3. 3Identify the pairs that trade against each other and decide which wins for which user segment.
  4. 4Measure every attribute on the same evaluation run so the trade is visible in one report.
  5. 5Set alerts where an attribute becomes a product failure, not where it becomes interesting.
  6. 6Revisit the targets when the model, data or traffic shape changes.

Step 1 of 6

List the attributes that matter for this product and the decision each one constrains.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. List the attributes that matter for this product and the decision each one constrains. 2. Give each a metric, a target and a tolerance: p95 latency, cost per request, refusal rate… 3. Identify the pairs that trade against each other and decide which wins for which user seg… 4. Measure every attribute on the same evaluation run so the trade is visible in one report. 5. Set alerts where an attribute becomes a product failure, not where it becomes interesting. 6. Revisit the targets when the model, data or traffic shape changes.

  1. List the attributes that matter for this product and the decision each one constrains.
  2. Give each a metric, a target and a tolerance: p95 latency, cost per request, refusal rate on a labelled set.
  3. Identify the pairs that trade against each other and decide which wins for which user segment.
  4. Measure every attribute on the same evaluation run so the trade is visible in one report.
  5. Set alerts where an attribute becomes a product failure, not where it becomes interesting.
  6. Revisit the targets when the model, data or traffic shape changes.

Trade-offs

ChoiceGood forCostFails when
Optimise latencyResponsive interfacesSmaller models or cached answers reduce qualityThe task needs multi-step reasoning
Optimise costSustainable unit economicsMore review and more failuresErrors are expensive to correct
Optimise safetyFewer harmful outputsLower coverage and more refusalsThe product serves legitimate sensitive content
Optimise explainabilityAuditable decisionsConstrained model choice and added workThe task is creative and open-ended
Trade-off mapInteractive

Good for

Responsive interfaces

Cost

Smaller models or cached answers reduce quality

Fails when

The task needs multi-step reasoning

  1. The task needs multi-step reasoning
  2. Errors are expensive to correct
  3. The product serves legitimate sensitive content
  4. The task is creative and open-ended

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Optimise latency is good for Responsive interfaces, costs Smaller models or cached answers reduce quality, and fails when The task needs multi-step reasoning. Optimise cost is good for Sustainable unit economics, costs More review and more failures, and fails when Errors are expensive to correct. Optimise safety is good for Fewer harmful outputs, costs Lower coverage and more refusals, and fails when The product serves legitimate sensitive content. Optimise explainability is good for Auditable decisions, costs Constrained model choice and added work, and fails when The task is creative and open-ended

Failure modes

  • Unmeasured attribute. A launch fails on latency or cost that nobody tracked. Debug: compare the pre-launch attribute list with the dashboards that actually exist.
  • Target without tolerance. Every fluctuation triggers a review and the team stops trusting alerts. Debug: check alert volume against the declared tolerance bands.
  • Conflict ignored. Safety filters and coverage are tuned separately until both regress. Debug: plot the pair on one chart for the same evaluation run.

In production

Report the attribute set on one dashboard per release and review the trade curve rather than single numbers. Alert on the attribute whose failure the user notices first.

Common misconceptions

People think: Quality means accuracy.

Actually: Accuracy without latency, cost and safety bounds describes a model, not a service.

People think: Attributes can be optimised independently.

Actually: They share one budget of model size, context and review effort; improving one usually charges another.

Interview angle

Strong answers translate each attribute into a measurement with a tolerance, then discuss the pairs that conflict and which one wins for which user. Weak answers list desirable properties without saying how any of them would be observed in production.

Sources