Skip to content
AI Engineer Guide.
juniorReportedcurated

When would you use a multimodal model instead of OCR plus an extraction pipeline, and what are its failure modes?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you control the cost of sending page images?
  • Budget tokens per page before choosing resolution: doubling resolution quadruples patch count, and tiling a large scan multiplies the bill by roughly the number of crops.
  • Use a fixed-query connector or a smaller resampler where detail is not needed, and crop the region of interest instead of sending the whole page.
  • Route pages through OCR when the task only needs text, and cache the stable prefix where the provider supports prompt caching.
How would you know the extraction is wrong without reading every page?
  • Build a per-field evaluation set stratified by document type and quality, and track field accuracy rather than whole-page similarity.
  • Add cross-field consistency checks such as totals equalling the sum of line items, and flag any page that fails before a human sees it.
  • Sample a fixed number of pages per batch for human review and compare the VLM's values against OCR bounding boxes on fields that carry money or identifiers.

Red flags

  • Treats the image as an opaque attachment with a flat token cost
  • Assumes VLM output is reliable for tables and numbers without verification

Concepts this tests

Tags

Source: engineeratlas.vercel.app

More in Multimodal, vision & voice