Why it matters
A model that answers correctly in a notebook can still be unusable: too slow for the interface, too expensive per request, wrong for a subset of users, or impossible to debug. Quality attributes make those constraints explicit and measurable, so design decisions can be argued with evidence rather than anecdotes, and the loudest demo stops winning the review.
Mental model
Quality attributes are the load limits on a bridge, not the bridge's length. The metaphor breaks because load limits are fixed by physics and certified, while these attributes are estimates that shift with data and must be re-measured as traffic changes.
How it works
- 1List the attributes that matter for this product and the decision each one constrains.
- 2Give each a metric, a target and a tolerance: p95 latency, cost per request, refusal rate on a labelled set.
- 3Identify the pairs that trade against each other and decide which wins for which user segment.
- 4Measure every attribute on the same evaluation run so the trade is visible in one report.
- 5Set alerts where an attribute becomes a product failure, not where it becomes interesting.
- 6Revisit the targets when the model, data or traffic shape changes.
Step 1 of 6
List the attributes that matter for this product and the decision each one constrains.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. List the attributes that matter for this product and the decision each one constrains. 2. Give each a metric, a target and a tolerance: p95 latency, cost per request, refusal rate… 3. Identify the pairs that trade against each other and decide which wins for which user seg… 4. Measure every attribute on the same evaluation run so the trade is visible in one report. 5. Set alerts where an attribute becomes a product failure, not where it becomes interesting. 6. Revisit the targets when the model, data or traffic shape changes.
- List the attributes that matter for this product and the decision each one constrains.
- Give each a metric, a target and a tolerance: p95 latency, cost per request, refusal rate on a labelled set.
- Identify the pairs that trade against each other and decide which wins for which user segment.
- Measure every attribute on the same evaluation run so the trade is visible in one report.
- Set alerts where an attribute becomes a product failure, not where it becomes interesting.
- Revisit the targets when the model, data or traffic shape changes.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Optimise latency | Responsive interfaces | Smaller models or cached answers reduce quality | The task needs multi-step reasoning |
| Optimise cost | Sustainable unit economics | More review and more failures | Errors are expensive to correct |
| Optimise safety | Fewer harmful outputs | Lower coverage and more refusals | The product serves legitimate sensitive content |
| Optimise explainability | Auditable decisions | Constrained model choice and added work | The task is creative and open-ended |
Good for
Responsive interfaces
Cost
Smaller models or cached answers reduce quality
Fails when
The task needs multi-step reasoning
- The task needs multi-step reasoning
- Errors are expensive to correct
- The product serves legitimate sensitive content
- The task is creative and open-ended
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Optimise latency is good for Responsive interfaces, costs Smaller models or cached answers reduce quality, and fails when The task needs multi-step reasoning. Optimise cost is good for Sustainable unit economics, costs More review and more failures, and fails when Errors are expensive to correct. Optimise safety is good for Fewer harmful outputs, costs Lower coverage and more refusals, and fails when The product serves legitimate sensitive content. Optimise explainability is good for Auditable decisions, costs Constrained model choice and added work, and fails when The task is creative and open-ended
Failure modes
- Unmeasured attribute. A launch fails on latency or cost that nobody tracked. Debug: compare the pre-launch attribute list with the dashboards that actually exist.
- Target without tolerance. Every fluctuation triggers a review and the team stops trusting alerts. Debug: check alert volume against the declared tolerance bands.
- Conflict ignored. Safety filters and coverage are tuned separately until both regress. Debug: plot the pair on one chart for the same evaluation run.
In production
Report the attribute set on one dashboard per release and review the trade curve rather than single numbers. Alert on the attribute whose failure the user notices first.
Common misconceptions
People think: Quality means accuracy.
Actually: Accuracy without latency, cost and safety bounds describes a model, not a service.
People think: Attributes can be optimised independently.
Actually: They share one budget of model size, context and review effort; improving one usually charges another.
Interview angle
Strong answers translate each attribute into a measurement with a tolerance, then discuss the pairs that conflict and which one wins for which user. Weak answers list desirable properties without saying how any of them would be observed in production.
Sources
- ISO/IEC 25010, Systems and software Quality Requirements and Evaluation — a standard vocabulary for quality attributes and their sub-characteristics.
- NIST, AI Risk Management Framework — trustworthiness characteristics and how they are measured.
- Breck et al., The ML Test Score — concrete checks that make quality claims testable.