juniorReportedcurated
When would you use a multimodal model instead of OCR plus an extraction pipeline, and what are its failure modes?
30-second answer
- Use the multimodal model when layout carries meaning, such as tables, forms, charts and scanned or photographed pages, because it reads the page as an image instead of reconstructing structure from extracted text.
- It collapses OCR, layout reconstruction and extraction into one call and handles cases where text extraction destroys the relationship between a label and its value.
- Keep deterministic OCR when you need exact character fidelity, auditability or very high volume, because image tokens scale with area and a 1024 by 1024 page becomes 4,096 visual tokens at 16-pixel patches.
- Its failure modes are dense tables where values drift between columns, small low-contrast text where it guesses, counting, and precise spatial relations, and it reports these as confident answers rather than difficulty.
- Cross-check numeric output against OCR or a parser and keep per-field provenance, because one call buys convenience and loses the coordinates that make an extraction auditable.
Likely follow-ups
Open a question to rehearse the answer.
How do you control the cost of sending page images?
- Budget tokens per page before choosing resolution: doubling resolution quadruples patch count, and tiling a large scan multiplies the bill by roughly the number of crops.
- Use a fixed-query connector or a smaller resampler where detail is not needed, and crop the region of interest instead of sending the whole page.
- Route pages through OCR when the task only needs text, and cache the stable prefix where the provider supports prompt caching.
How would you know the extraction is wrong without reading every page?
- Build a per-field evaluation set stratified by document type and quality, and track field accuracy rather than whole-page similarity.
- Add cross-field consistency checks such as totals equalling the sum of line items, and flag any page that fails before a human sees it.
- Sample a fixed number of pages per batch for human review and compare the VLM's values against OCR bounding boxes on fields that carry money or identifiers.
Red flags
- Treats the image as an opaque attachment with a flat token cost
- Assumes VLM output is reliable for tables and numbers without verification
Concepts this tests
Tags
Source: engineeratlas.vercel.app
More in Multimodal, vision & voice
- Argue both sides: cascaded ASR plus LLM plus TTS versus a native speech-to-speech model for a production voice agent.
- Budget the latency for a real-time voice agent: VAD, ASR, LLM, TTS, network. Where does the time go, and what do you optimise first?
- A video summariser samples one frame every three seconds. A customer summary says an export may not have finished, but the recording shows an Export ready toast. What went wrong, and how do you fix it?