Why it matters
A detector tells you a cup is present, not that it is behind the kettle or moving toward the edge of the frame. Robotics, augmented reality and video analytics need geometry and change over time. Each is estimated from cues that are individually ambiguous: one view cannot distinguish a small near object from a large far one, and apparent motion mixes camera movement with object movement.
Mental model
Spatial understanding is reconstructing a room from a photograph: you infer depth from shading and perspective, and you can be confidently wrong. The metaphor breaks because a model can also use learned priors about object sizes, which sometimes beats geometry and sometimes overrides it.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic room drawn from a class map, with authored depth values and two motion-vector fields: one from a camera pan, one after subtracting ego-motion. No depth model, flow model or tracker was run, and no metric scale is encoded.
What to notice. A box gives image position, not distance. The depth map orders the cup in front of the table and the kettle behind it, with no metres. Optical flow from a panning camera makes every stationary object appear to move; subtracting the camera leaves the cup as the only mover. A depth map that is accurate per frame can still be unusable if it flickers.
Text version
A schematic room with a window, lamp, table, chair, cup and kettle, drawn as a 24 by 16 grid. Steps show detection boxes for the cup, kettle and table with authored confidences, a depth map with the cup nearest and the wall furthest, a note that one view is scale-ambiguous, a camera-pan field where static objects move 1.9 cells and the cup moves 2.3, and an ego-motion-subtracted field where the cup's residual is 0.4 cells and the static objects stay near zero. Nearer cells carry more colour.
- Separate the questions: relative position in the image, depth ordering, metric distance, or velocity in the world.
- Choose a representation that matches the output, such as boxes with depth order, a depth map, or tracked trajectories.
- Fix the camera model, since intrinsics and distortion change every geometric estimate downstream.
- For motion, distinguish ego-motion from object motion by estimating camera movement first.
- Evaluate with task-specific metrics and a temporal consistency check, not only per-frame error.
- Test on the capture conditions you will see: wide angle, low light, rolling shutter and compression.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Monocular depth | Single camera, cheap | Scale ambiguity and reflective surfaces | Metric distance is required |
| Stereo or depth sensor | Real scale | Hardware, calibration and range limits | Devices are commodity cameras |
| Optical flow | Dense motion cues | Apparent motion only | The camera moves |
| Object tracking | Identity over time | Drift and occlusion | Objects leave and re-enter |
Good for
Single camera, cheap
Cost
Scale ambiguity and reflective surfaces
Fails when
Metric distance is required
- Metric distance is required
- Devices are commodity cameras
- The camera moves
- Objects leave and re-enter
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Monocular depth is good for Single camera, cheap, costs Scale ambiguity and reflective surfaces, and fails when Metric distance is required. Stereo or depth sensor is good for Real scale, costs Hardware, calibration and range limits, and fails when Devices are commodity cameras. Optical flow is good for Dense motion cues, costs Apparent motion only, and fails when The camera moves. Object tracking is good for Identity over time, costs Drift and occlusion, and fails when Objects leave and re-enter
Failure modes
- Scale ambiguity. Distances are consistent within a frame and wrong across frames. Debug: compare predicted depth with a known object size or a measured baseline in a few scenes.
- Ego-motion misread as object motion. Everything appears to move when the camera pans. Debug: estimate global motion and subtract it, then check whether stationary objects report near-zero velocity.
- Temporal flicker. Depth or tracks jitter between frames and downstream logic oscillates. Debug: plot a per-frame metric over a static clip and look for high-frequency variation.
In production
Monitor temporal consistency and calibration drift, since a bumped camera invalidates geometry silently. Rehearse with the actual device and mounting before trusting metric outputs.
Common misconceptions
People think: Depth from one image is a measurement.
Actually: It is an estimate with scale ambiguity unless the scene or a prior fixes scale.
People think: Optical flow gives object velocity.
Actually: It gives apparent pixel motion, which includes camera movement and rotation.
Interview angle
Separate depth ordering from metric distance and ego-motion from object motion, since conflating them is the usual mistake. A strong answer names calibration and temporal consistency as production concerns. Weak answers quote a depth model's accuracy without saying what fixes scale.
Sources
- Ranftl et al., Towards Robust Monocular Depth Estimation — MiDaS and mixed-dataset monocular depth.
- Teed & Deng, RAFT: Recurrent All-Pairs Field Transforms for Optical Flow — dense optical flow estimation.
- Yang et al., Depth Anything — large-scale monocular depth with strong generalisation.