Skip to content
AI Engineer Guide.

concept · concepts · advanced · concept 18 of 22

Spatial & motion

0 of 22 complete

Why it matters

A detector tells you a cup is present, not that it is behind the kettle or moving toward the edge of the frame. Robotics, augmented reality and video analytics need geometry and change over time. Each is estimated from cues that are individually ambiguous: one view cannot distinguish a small near object from a large far one, and apparent motion mixes camera movement with object movement.

Mental model

Spatial understanding is reconstructing a room from a photograph: you infer depth from shading and perspective, and you can be confidently wrong. The metaphor breaks because a model can also use learned priors about object sizes, which sometimes beats geometry and sometimes overrides it.

How it works

Position, depth, and apparent motionInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic room drawn from a class map, with authored depth values and two motion-vector fields: one from a camera pan, one after subtracting ego-motion. No depth model, flow model or tracker was run, and no metric scale is encoded.

What to notice. A box gives image position, not distance. The depth map orders the cup in front of the table and the kettle behind it, with no metres. Optical flow from a panning camera makes every stationary object appear to move; subtracting the camera leaves the cup as the only mover. A depth map that is accurate per frame can still be unusable if it flickers.

Text version

A schematic room with a window, lamp, table, chair, cup and kettle, drawn as a 24 by 16 grid. Steps show detection boxes for the cup, kettle and table with authored confidences, a depth map with the cup nearest and the wall furthest, a note that one view is scale-ambiguous, a camera-pan field where static objects move 1.9 cells and the cup moves 2.3, and an ego-motion-subtracted field where the cup's residual is 0.4 cells and the static objects stay near zero. Nearer cells carry more colour.

  1. Separate the questions: relative position in the image, depth ordering, metric distance, or velocity in the world.
  2. Choose a representation that matches the output, such as boxes with depth order, a depth map, or tracked trajectories.
  3. Fix the camera model, since intrinsics and distortion change every geometric estimate downstream.
  4. For motion, distinguish ego-motion from object motion by estimating camera movement first.
  5. Evaluate with task-specific metrics and a temporal consistency check, not only per-frame error.
  6. Test on the capture conditions you will see: wide angle, low light, rolling shutter and compression.

Trade-offs

ChoiceGood forCostFails when
Monocular depthSingle camera, cheapScale ambiguity and reflective surfacesMetric distance is required
Stereo or depth sensorReal scaleHardware, calibration and range limitsDevices are commodity cameras
Optical flowDense motion cuesApparent motion onlyThe camera moves
Object trackingIdentity over timeDrift and occlusionObjects leave and re-enter
Trade-off mapInteractive

Good for

Single camera, cheap

Cost

Scale ambiguity and reflective surfaces

Fails when

Metric distance is required

  1. Metric distance is required
  2. Devices are commodity cameras
  3. The camera moves
  4. Objects leave and re-enter

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Monocular depth is good for Single camera, cheap, costs Scale ambiguity and reflective surfaces, and fails when Metric distance is required. Stereo or depth sensor is good for Real scale, costs Hardware, calibration and range limits, and fails when Devices are commodity cameras. Optical flow is good for Dense motion cues, costs Apparent motion only, and fails when The camera moves. Object tracking is good for Identity over time, costs Drift and occlusion, and fails when Objects leave and re-enter

Failure modes

  • Scale ambiguity. Distances are consistent within a frame and wrong across frames. Debug: compare predicted depth with a known object size or a measured baseline in a few scenes.
  • Ego-motion misread as object motion. Everything appears to move when the camera pans. Debug: estimate global motion and subtract it, then check whether stationary objects report near-zero velocity.
  • Temporal flicker. Depth or tracks jitter between frames and downstream logic oscillates. Debug: plot a per-frame metric over a static clip and look for high-frequency variation.

In production

Monitor temporal consistency and calibration drift, since a bumped camera invalidates geometry silently. Rehearse with the actual device and mounting before trusting metric outputs.

Common misconceptions

People think: Depth from one image is a measurement.

Actually: It is an estimate with scale ambiguity unless the scene or a prior fixes scale.

People think: Optical flow gives object velocity.

Actually: It gives apparent pixel motion, which includes camera movement and rotation.

Interview angle

Separate depth ordering from metric distance and ego-motion from object motion, since conflating them is the usual mistake. A strong answer names calibration and temporal consistency as production concerns. Weak answers quote a depth model's accuracy without saying what fixes scale.

Sources