Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 50 of 57

Multimodal RAG

0 of 57 complete

Why it matters

Technical documents carry meaning in figures, tables and scanned pages that text extraction flattens or drops. Retrieval over text alone cannot answer a question whose evidence is a chart. Multimodal RAG extends the index to those artefacts, and the design question is how to make an image comparable to a text query: embed both into one space, or generate a text surrogate and index that.

Mental model

Multimodal RAG is a librarian who can search by picture as well as by title. The metaphor breaks because the picture search matches visual and topical similarity, not the specific value printed inside the chart, so a figure about revenue can be returned for a question about costs.

How it works

What a text index losesInteractive

Synthetic annual report · page 14

One figure, one table, one caption

bounding boxes

Step 1/6 · Text extraction keeps the prose

Step 1Text extraction keeps the prose

A text extractor walks the text layer. It returns the title, paragraphs, caption and table cells, but the bars in the figure are pixels with no text inside them.

NORTHWIND · ANNUAL REPORT 2025 · PAGE 14

, running header, region 1. Select to inspect.

Revenue by plan

, title, region 2. Select to inspect.
Text

Enterprise was the fastest-growing plan in FY2025, adding $13m while Business added $7m and Starter added $2m.

, paragraph, region 3. Select to inspect.
FIGURE 3
EntBusSta
∅, figure FIGURE 3, region 4, not extracted. Select to inspect.

Figure 3. Revenue by plan, FY2025 ($m). Source: synthetic ledger roll-up.

E, caption, region 5, extracted. Select to inspect.
TABLE 2
PlanFY25
Enterprise74
Business52
Starter21
, table TABLE 2, region 6. Select to inspect.
Text

Enterprise includes multi-year contracts. Figures are rounded to the nearest $1m.

, paragraph, region 7. Select to inspect.
Retrieval hitfigure 3 + caption + table row · page 14
, form field Retrieval hit, region 9. Select to inspect.

page render 1240 × 1754 · OCR text indexed beside the image

, running footer, region 10. Select to inspect.
9 regions

Step record

Extracted
prose + table
Figure values
missed
Caption
present, not the values

The text version summarises this figure; select a region for its full text.

Marksnot extractedextracted

Figure. What a text index drops and a multimodal index keeps: the text view sees the caption but not the bars, while the multimodal view keeps the figure, its caption and the table as retrievable artefacts. Bar heights are drawn from the table's FY2025 values.

What to notice. Synthetic report page. Bar heights are drawn from the adjacent table; the caption text is authored, in the position a generated caption would occupy. The figure shows what each index can carry, not retrieval accuracy.

Text version

A synthetic annual-report page titled Revenue by plan. A bar chart shows three bars: enterprise 74, business 52 and starter 21 in millions. A caption below names the figure, its units and the source. A table to the right repeats the same three values. An index-entries block lists two text chunks, one image reference with a page crop, one caption entry and one table entry. The first step marks the figure as missing, because a text extractor has no text for the bars; the second marks it extracted after a caption is generated. A chunk overlay groups the prose, the figure and caption, and the table into separate chunks. A retrieval-hit field names the figure, caption and table row returned for one query. Selecting any region shows its text, type, state and chunk.

  1. Classify the corpus by modality and decide per type whether to embed natively, caption, or keep both.
  2. For figures and charts, generate a description that includes the axes, units and the trend, then index it with the image reference.
  3. For scanned pages, index the page image with OCR text so a hit can be rendered for the generator.
  4. Align modalities in one space only if the model supports it; otherwise keep separate indexes and fuse results.
  5. Attach the image reference and page number to each chunk so the generator can be given the actual figure.
  6. Evaluate per modality, since text recall says nothing about whether a chart was retrieved correctly.

Trade-offs

ChoiceGood forCostFails when
Joint image-text embeddingCross-modal queries in one indexWeaker fine-grained text matchingQuestions ask for a specific value in a figure
Caption then text indexCheap and reuses the text stackCaption errors become retrieval errorsThe figure's meaning is visual, not verbal
Page images to a vision modelPreserves layout and tablesHigh token cost and latencyVolume is high or answers are simple
Text onlySimplest pipelineMisses figures and scanned contentThe corpus is reports or slides
Trade-off mapInteractive

Good for

Cross-modal queries in one index

Cost

Weaker fine-grained text matching

Fails when

Questions ask for a specific value in a figure

  1. Questions ask for a specific value in a figure
  2. The figure's meaning is visual, not verbal
  3. Volume is high or answers are simple
  4. The corpus is reports or slides

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Joint image-text embedding is good for Cross-modal queries in one index, costs Weaker fine-grained text matching, and fails when Questions ask for a specific value in a figure. Caption then text index is good for Cheap and reuses the text stack, costs Caption errors become retrieval errors, and fails when The figure's meaning is visual, not verbal. Page images to a vision model is good for Preserves layout and tables, costs High token cost and latency, and fails when Volume is high or answers are simple. Text only is good for Simplest pipeline, costs Misses figures and scanned content, and fails when The corpus is reports or slides

Failure modes

  • Caption hallucination. The generated description invents a trend and the figure becomes retrievable for a claim it does not make. Debug: compare the caption with the rendered image for a sample of figures.
  • Modality mismatch in ranking. Image and text scores are fused without calibration, so one modality always dominates. Debug: print the score ranges per modality for a mixed query.
  • Unrenderable hit. The retrieved image cannot be shown to the generator because only a reference was stored. Debug: resolve every image reference in a sample of answers and confirm the asset exists.

In production

Track retrieval rate per modality and caption quality on a sample. Cap image tokens per request and cache rendered pages, since repeated rendering dominates latency.

Common misconceptions

People think: A vision model makes OCR unnecessary.

Actually: Vision models help with layout and hard scans but are costly and less deterministic than a text layer when one exists.

People think: One joint embedding handles every modality equally.

Actually: Cross-modal spaces are uneven; fine-grained values in charts often remain easier to retrieve through text.

Interview angle

Choose between joint embedding and text surrogates per modality, and explain what each loses. Cover rendering the artefact for the generator and per-modality evaluation. Embedding images with CLIP, with no discussion of captions, layout or token cost, is the thin answer.

Sources