Why it matters
A single label per image is the simplest vision task and the least useful for most products. Real requests ask where an object is, how many there are, or what is written on it. Each answer format needs its own head, loss and metric, and a model trained for one will produce confident output in the shape of another. Choosing the task shape comes before choosing a model.
Mental model
Image understanding is describing a room to someone who cannot see it: the useful description depends on what they need to do next. The metaphor breaks because a model's description is generated from features, not from attention to your purpose, so it may be fluent and irrelevant.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic street corner drawn from a class map, with four output contracts: one class, four boxes, a pixel mask, or a description. Boxes, confidences and the coarse 3 × 2 patch grid at the last step are authored illustrations; no photograph or model output is used.
What to notice. The contract decides the metric and the failure. A caption can be fluent about a sign whose box is the least confident. The last step collapses the sign into one 8-cell patch: at that resolution the detail is gone before the model sees it, and aggregate accuracy will not show it.
Text version
A schematic street corner with sky, a building, a tree, a road, a car, a person and a shop sign. Steps show the four output contracts, a whole-frame classification, four detection boxes with authored confidences (car 0.93, person 0.88, sign 0.71, tree 0.64), a segmentation mask for those four objects, a caption, and a 3 by 2 patch grid in which the sign occupies one patch. A patch-token slider maps each patch to its grid position.
- Write the output contract first: a class, a set of boxes with labels, pixel masks, or free text.
- Match the metric to the contract, using per-class recall for detection and overlap-based measures for masks.
- Check the smallest object you must detect against the input resolution, since a downscaled image loses it.
- Build a slice of hard cases: occlusion, unusual lighting, small instances and near-duplicate classes.
- For descriptive outputs, evaluate factuality against the image rather than fluency of the sentence.
- Inspect errors visually, because a confusion matrix cannot show that a box is offset by twenty pixels.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Classifier | Cheap, fast, easy to monitor | No location or count | The product needs positions |
| Detector | Boxes and counts | Annotation cost and crowded-scene errors | Objects overlap heavily |
| Segmentation | Pixel-accurate masks | Expensive labels and compute | You only need presence |
| Vision-language description | Flexible, no fixed taxonomy | Hallucination and no calibration | You need counts or coordinates |
Good for
Cheap, fast, easy to monitor
Cost
No location or count
Fails when
The product needs positions
- The product needs positions
- Objects overlap heavily
- You only need presence
- You need counts or coordinates
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Classifier is good for Cheap, fast, easy to monitor, costs No location or count, and fails when The product needs positions. Detector is good for Boxes and counts, costs Annotation cost and crowded-scene errors, and fails when Objects overlap heavily. Segmentation is good for Pixel-accurate masks, costs Expensive labels and compute, and fails when You only need presence. Vision-language description is good for Flexible, no fixed taxonomy, costs Hallucination and no calibration, and fails when You need counts or coordinates
Failure modes
- Downscale blindness. Small objects disappear before the model sees them. Debug: compare detections on the original and the resized tensor for images known to contain small instances.
- Taxonomy confusion. Two visually similar classes are swapped consistently. Debug: inspect the confusion matrix and review the top confused pairs by hand.
- Description hallucination. The caption mentions objects that are absent. Debug: check each mentioned entity against detections or a human-labelled sample.
In production
Monitor per-class recall and the distribution of predicted counts, since a collapse to one class shows up there first. Keep a human-reviewed sample per class for drift.
Common misconceptions
People think: A caption proves the model saw the detail.
Actually: Generated text can be plausible without being grounded; verify entities against detections or labels.
People think: Higher accuracy means a better product.
Actually: The metric may not measure the error that matters, such as localisation or count.
Interview angle
Choose the output contract before the model, and pick a metric that can express the error you care about. A strong answer distinguishes a box offset from a missed detection. Weak answers quote classifier accuracy for a task that needs counting or localisation.
Sources
- Dosovitskiy et al., An Image is Worth 16x16 Words — the backbone used across classification and detection variants.
- He et al., Deep Residual Learning for Image Recognition — residual backbones that made deep vision models trainable.
- Radford et al., CLIP — open-vocabulary classification from image-text pairs.