Why it matters
Technical documents carry meaning in figures, tables and scanned pages that text extraction flattens or drops. Retrieval over text alone cannot answer a question whose evidence is a chart. Multimodal RAG extends the index to those artefacts, and the design question is how to make an image comparable to a text query: embed both into one space, or generate a text surrogate and index that.
Mental model
Multimodal RAG is a librarian who can search by picture as well as by title. The metaphor breaks because the picture search matches visual and topical similarity, not the specific value printed inside the chart, so a figure about revenue can be returned for a question about costs.
How it works
Synthetic annual report · page 14
One figure, one table, one caption
bounding boxes
Step 1Text extraction keeps the prose
A text extractor walks the text layer. It returns the title, paragraphs, caption and table cells, but the bars in the figure are pixels with no text inside them.
NORTHWIND · ANNUAL REPORT 2025 · PAGE 14
Revenue by plan
Figure 3. Revenue by plan, FY2025 ($m). Source: synthetic ledger roll-up.
| Plan | FY25 |
|---|---|
| Enterprise | 74 |
| Business | 52 |
| Starter | 21 |
page render 1240 × 1754 · OCR text indexed beside the image
Step record
- Extracted
- prose + table
- Figure values
- missed
- Caption
- present, not the values
The text version summarises this figure; select a region for its full text.
Figure. What a text index drops and a multimodal index keeps: the text view sees the caption but not the bars, while the multimodal view keeps the figure, its caption and the table as retrievable artefacts. Bar heights are drawn from the table's FY2025 values.
What to notice. Synthetic report page. Bar heights are drawn from the adjacent table; the caption text is authored, in the position a generated caption would occupy. The figure shows what each index can carry, not retrieval accuracy.
Text version
A synthetic annual-report page titled Revenue by plan. A bar chart shows three bars: enterprise 74, business 52 and starter 21 in millions. A caption below names the figure, its units and the source. A table to the right repeats the same three values. An index-entries block lists two text chunks, one image reference with a page crop, one caption entry and one table entry. The first step marks the figure as missing, because a text extractor has no text for the bars; the second marks it extracted after a caption is generated. A chunk overlay groups the prose, the figure and caption, and the table into separate chunks. A retrieval-hit field names the figure, caption and table row returned for one query. Selecting any region shows its text, type, state and chunk.
- Classify the corpus by modality and decide per type whether to embed natively, caption, or keep both.
- For figures and charts, generate a description that includes the axes, units and the trend, then index it with the image reference.
- For scanned pages, index the page image with OCR text so a hit can be rendered for the generator.
- Align modalities in one space only if the model supports it; otherwise keep separate indexes and fuse results.
- Attach the image reference and page number to each chunk so the generator can be given the actual figure.
- Evaluate per modality, since text recall says nothing about whether a chart was retrieved correctly.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Joint image-text embedding | Cross-modal queries in one index | Weaker fine-grained text matching | Questions ask for a specific value in a figure |
| Caption then text index | Cheap and reuses the text stack | Caption errors become retrieval errors | The figure's meaning is visual, not verbal |
| Page images to a vision model | Preserves layout and tables | High token cost and latency | Volume is high or answers are simple |
| Text only | Simplest pipeline | Misses figures and scanned content | The corpus is reports or slides |
Good for
Cross-modal queries in one index
Cost
Weaker fine-grained text matching
Fails when
Questions ask for a specific value in a figure
- Questions ask for a specific value in a figure
- The figure's meaning is visual, not verbal
- Volume is high or answers are simple
- The corpus is reports or slides
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Joint image-text embedding is good for Cross-modal queries in one index, costs Weaker fine-grained text matching, and fails when Questions ask for a specific value in a figure. Caption then text index is good for Cheap and reuses the text stack, costs Caption errors become retrieval errors, and fails when The figure's meaning is visual, not verbal. Page images to a vision model is good for Preserves layout and tables, costs High token cost and latency, and fails when Volume is high or answers are simple. Text only is good for Simplest pipeline, costs Misses figures and scanned content, and fails when The corpus is reports or slides
Failure modes
- Caption hallucination. The generated description invents a trend and the figure becomes retrievable for a claim it does not make. Debug: compare the caption with the rendered image for a sample of figures.
- Modality mismatch in ranking. Image and text scores are fused without calibration, so one modality always dominates. Debug: print the score ranges per modality for a mixed query.
- Unrenderable hit. The retrieved image cannot be shown to the generator because only a reference was stored. Debug: resolve every image reference in a sample of answers and confirm the asset exists.
In production
Track retrieval rate per modality and caption quality on a sample. Cap image tokens per request and cache rendered pages, since repeated rendering dominates latency.
Common misconceptions
People think: A vision model makes OCR unnecessary.
Actually: Vision models help with layout and hard scans but are costly and less deterministic than a text layer when one exists.
People think: One joint embedding handles every modality equally.
Actually: Cross-modal spaces are uneven; fine-grained values in charts often remain easier to retrieve through text.
Interview angle
Choose between joint embedding and text surrogates per modality, and explain what each loses. Cover rendering the artefact for the generator and per-modality evaluation. Embedding images with CLIP, with no discussion of captions, layout or token cost, is the thin answer.
Sources
- Radford et al., CLIP — contrastive image-text representations that enable cross-modal search.
- Liu et al., LLaVA — visual instruction tuning for models that read images in a prompt.
- Huang et al., LayoutLMv3 — joint text and layout modelling for document images.