Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 1 of 22

Multimodal models

0 of 22 complete

Why this is a systems decision

Most organisational data is not prose. It is invoices, dashboards, screenshots, scanned forms, product photos, recorded calls. A text-only pipeline has to convert all of it into text before the model sees anything, and every conversion step loses something: table structure, reading order, the difference between a label and a value.

Multimodal models move the conversion inside the model. That is a genuine capability change — a model can relate a chart's axis to its legend and to the question — but it is not free. Images consume context, prefill time and money in proportion to their resolution, and the model's reading of a document is a prediction, not an extraction with coordinates.

The engineering question is therefore not "can the model see images" but "which parts of this task need perception, and which need deterministic extraction".

Mental model

A translator sitting beside a colleague who cannot see the slide. The colleague can ask questions, and the translator describes what matters — but only what the translator noticed and chose to mention.

The analogy breaks in the mechanism and in the failure mode. Modern models do not convert the image to a caption first: patches become tokens in the same sequence, and attention relates them directly to the question, so detail the model never verbalises can still influence the answer. And because perception is trained jointly with generation, the model can produce a plausible description of detail that is not there — a failure a human translator rarely makes.

How it works

The same invoice at three patch sizesInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic invoice drawn at 24 × 32 cells standing in for 384 × 512 pixels: 768 visual tokens at 16 px patches, 192 at 32 px and 48 at 64 px, where the invoice number is a single token. Two authored field boxes carry the coordinates a sentence does not, and the mask selects the fields to score.

What to notice. Resolution is the budget and the legibility limit at once. Doubling the patch quarters the tokens and merges pairs of digits; at 64 px the whole invoice number is one token with no characters in it. OCR returns a box, a coordinate and a confidence per field, which is the artefact to audit; page-level accuracy averages the one misread total away.

Text version

A schematic invoice with a heading, logo, a four-column table, a total and an invoice number. Steps show 768 visual tokens at 16-pixel patches, 192 at 32-pixel patches, 48 at 64-pixel patches where the invoice number collapses to token 8, two detection boxes labelled invoice no. 0.97 and total 0.95, and a mask covering only those two fields. A patch-token slider reports row and column, zero-based.

  1. Preprocess. Decode the image, convert colour space, resize or tile to the encoder's expected input, and normalise pixel values. Cropping and tiling decisions here determine what the model can possibly read.
  2. Patchify. Split the image into fixed-size patches, flatten each patch, and project it linearly into an embedding. A 1024×1024 image with 16-pixel patches becomes a 64×64 grid of 4,096 visual tokens before any compression.
  3. Encode. A vision transformer processes the patch sequence with self-attention, producing contextual features rather than independent patch vectors.
  4. Connect. A connector maps vision features into the language model's embedding space. Common designs are a small MLP, a resampler that compresses many patches into a fixed number of queries, or gated cross-attention layers interleaved through the language model.
  5. Interleave. Visual embeddings are inserted into the token sequence at the position of an image placeholder, so a single causal decoder attends over text and image tokens together. Dual-stream designs keep the modalities separate and cross-attend instead.
  6. Generate. The model emits text tokens autoregressively, conditioned on the joint sequence. Structured output constraints apply exactly as they do for text.
  7. Train in stages. Contrastive image–text pretraining aligns a vision encoder with language; connector training teaches the mapping on caption pairs; visual instruction tuning trains the combined model on image–instruction–answer triples. Early-fusion models instead tokenise images with a discrete tokeniser and train one transformer over mixed tokens, which also allows image generation but costs more compute.

The token arithmetic is the part teams underestimate:

def visual_tokens(height: int, width: int, patch: int = 16, resampler: int | None = None) -> int:
    patches = (height // patch) * (width // patch)
    if resampler:
        return resampler   # fixed-query connector: constant cost, less detail
    return patches
 
for h, w in [(768, 768), (1024, 1024), (2048, 1536)]:
    print(h, w, visual_tokens(h, w), visual_tokens(h, w, resampler=256))

Doubling resolution quadruples patch count. Tiling a large scan into four crops multiplies the token bill by roughly four, which is why "just use a higher resolution" is a cost decision as much as a quality one. Fixed-query connectors cap the cost but discard detail.

Trade-offs

ApproachGood forCostFails when
General VLM, directMixed documents, photos, ad-hoc questionsImage tokens, lower precision on dense textYou need exact field values with coordinates
OCR plus layout, then text LLMDense forms, tables, audit requirementsPipeline complexity, layout heuristicsThe question is about visual relationships
VLM with OCR fallbackProduction document intakeTwo systems, routing logicRouting confidence is unmeasured
Specialised document modelOne high-volume document typeTraining data, narrow scopeThe document mix changes
Trade-off mapInteractive

Good for

Mixed documents, photos, ad-hoc questions

Cost

Image tokens, lower precision on dense text

Fails when

You need exact field values with coordinates

  1. You need exact field values with coordinates
  2. The question is about visual relationships
  3. Routing confidence is unmeasured
  4. The document mix changes

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: General VLM, direct is good for Mixed documents, photos, ad-hoc questions, costs Image tokens, lower precision on dense text, and fails when You need exact field values with coordinates. OCR plus layout, then text LLM is good for Dense forms, tables, audit requirements, costs Pipeline complexity, layout heuristics, and fails when The question is about visual relationships. VLM with OCR fallback is good for Production document intake, costs Two systems, routing logic, and fails when Routing confidence is unmeasured. Specialised document model is good for One high-volume document type, costs Training data, narrow scope, and fails when The document mix changes

Failure modes

  • Small text at low resolution. The answer is wrong because the text was never legible after resize. Debug by cropping the region at full resolution and re-asking; if accuracy jumps, the problem is preprocessing, not reasoning.
  • Counting and spatial relations. The model says four columns when there are five, or confuses left and right. Debug by asking for a structured enumeration with locations, and cross-check against a deterministic detector or OCR bounding boxes.
  • Hallucinated detail. The model describes a table cell that is empty or a logo that is not present. Debug by requiring verbatim quotes with bounding boxes, and by testing on images with deliberately blank regions.
  • Aspect-ratio distortion. Non-uniform resizing changes text shape and hurts recognition. Debug by logging original and resized dimensions per request and by testing the same document at several aspect ratios.
  • Instructions inside images. A screenshot contains text telling the model to ignore its instructions. Debug by tracing which image was attached when behaviour changed; treat image text as untrusted input.

In production

Budget tokens per image and per request before choosing a model. A document pipeline that attaches ten pages at high resolution can spend more on prefill than on the entire answer, and the cost is invisible unless you log image token counts separately from text.

Make preprocessing deterministic and versioned: resize policy, tiling grid, order of pages, and colour handling. When quality changes after a deploy, the first thing to compare is the exact bytes and dimensions the model received, not the prompt.

Build an evaluation set of real documents with per-field expectations, stratified by document type and quality. Aggregate accuracy hides the failure that matters — the one field, on the one form, that is always misread.

Keep a deterministic path for anything that must be exact. OCR with coordinates and a parser is auditable in a way a VLM's reading is not; using the VLM to interpret and the OCR to verify is often the strongest combination.

Common misconceptions

People think: A multimodal model performs OCR internally, so you can drop your OCR pipeline.

Actually: It reads text approximately, without coordinates or per-character confidence. For exact extraction, audit trails or dense forms, deterministic OCR remains the better primitive.

People think: Higher resolution always improves accuracy.

Actually: Resolution helps small text, then hits diminishing returns while token cost grows quadratically. Past the point where the text is legible, more pixels mostly buy cost.

People think: Images are a safe input channel.

Actually: Images carry text, and text inside an image is untrusted content that reaches the model's context. It needs the same injection defences as a fetched web page.

Interview angle

"Design a pipeline that extracts line items from supplier invoices." The strong answer does not start with a model. It states the accuracy requirement and audit need, proposes OCR plus layout for exact fields with a VLM for interpretation and exceptions, defines routing and fallback, and budgets image tokens per page. Then it names the failure to instrument: a field that is consistently misread at low resolution. See ASR and the multimodal interview category.

Sources