Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 4 of 22

Document understanding

0 of 22 complete

Why it matters

A PDF is a rendering, not a data structure. The same text appears in a header, a footnote and a table cell, and only position distinguishes them. Pipelines that flatten a page to a string lose the relationships that make an invoice an invoice. Document understanding exists to recover those relationships, and its failures are expensive because they surface as wrong numbers downstream.

Mental model

Document understanding is rebuilding a form from a photograph: you need the boxes and their positions, not only the words inside them. The metaphor breaks because a model can learn layout conventions, so it can infer a field's role from position even when the label above it is ambiguous.

How it works

Fields, confidence and arithmeticInteractive

Synthetic supplier invoice · page 1 of 1

Five fields, one line-item table, one validation rule

bounding boxes

Step 1/6 · Regions before fields

Step 1Regions before fields

The invoice number sits top-right, the vendor top-left, line items in the middle and the total at the bottom right. Position is part of the label: the same string can appear in several roles.

NORTHWIND SUPPLIES · INVOICE · PAGE 1 OF 1

, running header, region 1. Select to inspect.

Invoice NW-8841

, title, region 2. Select to inspect.
Invoice no.NW-8841
99, form field Invoice no., region 3, confidence 99%. Select to inspect.
Date2026-10-02
98, form field Date, region 4, confidence 98%. Select to inspect.
VendorNorthwind Supplies GmbH
97, form field Vendor, region 5, confidence 97%. Select to inspect.
PO4471-A
91, form field PO, region 6, confidence 91%. Select to inspect.
Due2026-11-01
96, form field Due, region 7, confidence 96%. Select to inspect.
LINE ITEMS
DescriptionQtyUnitAmount
Valve assembly12640.007,680.00
Seal kit4048.00!1,820.00
Freight1380.00380.00
Subtotal9,880.00
Tax2,500.00
Total✕12,480.00
93, table LINE ITEMS, region 8, confidence 93%. Select to inspect.
Arithmetic check9,880.00 + 2,500.00 = 12,380.00 ≠ 12,480.00
✕, form field Arithmetic check, region 9, check failed. Select to inspect.
Review queueline 2 amount · 100.00 difference
?, form field Review queue, region 10, needs review. Select to inspect.
Text

Rule: when a line item is below the confidence floor and the printed total disagrees with the parts, route the page; do not pick a total.

?, paragraph, region 11, needs review. Select to inspect.
EXTRACTION RECORD
fields 5 · table cells 24 · low 1 route review · reason total_mismatch page image kept · offsets stored reviewer sees line 2 and the difference
, code block EXTRACTION RECORD, region 12. Select to inspect.

extracted 2026-11-17 09:41 · 5 fields · 1 line item low · 1 page routed

?, running footer, region 13, needs review. Select to inspect.
13 regions

Step record

Document type
invoice
Target
fields + table
Page
1 of 1

The text version summarises this figure; select a region for its full text.

Markspasslow confidencecheck failedneeds review

Figure. An invoice page as a field-extraction target: vendor, dates, line items and total are located by layout, read with confidence and validated by arithmetic. One misread line-item amount makes the total check fail, which routes the page to review instead of accepting a plausible number.

What to notice. Synthetic invoice. All values, coordinates and confidences are authored; the arithmetic is exact for the numbers shown. The figure demonstrates a validation rule, not a model's measured field accuracy.

Text version

A synthetic supplier invoice. Fields hold the invoice number, date, vendor, purchase order and due date. A six-row line-item table lists valve assemblies, seal kits, freight, subtotal, tax and total. The seal-kit amount reads 1,820.00 at 78 percent confidence; the line items sum to 9,880.00 and with 2,500.00 tax give 12,380.00 against a printed total of 12,480.00, so the total row is marked failed. A validation block shows the mismatch, a review block records the field and the 100.00 difference, and an extraction record gives the field count, the low cell and the review route. Steps show bounding boxes, confidence, the rebuilt table, the arithmetic failure, the review route and the per-field evaluation.

  1. Classify the document types in scope, since invoices, contracts and scientific papers need different schemas.
  2. Decide the extraction target: plain text with reading order, key-value fields, or full table structure.
  3. Choose the pipeline shape, either OCR plus a layout model or an end-to-end model that reads page images.
  4. Preserve coordinates through the pipeline so every extracted value can be traced to a region on the page.
  5. Validate extracted fields with type and range checks, and route failures to review rather than guessing.
  6. Evaluate per field and per document type, because an average over pages hides the field that is always wrong.

Trade-offs

ChoiceGood forCostFails when
OCR plus rulesTransparent and cheapBrittle across layoutsTemplates change often
Layout-aware modelHandles varied layoutsNeeds labelled pagesDocuments are handwritten or scanned poorly
End-to-end vision modelNo OCR stage to maintainHallucinated fields and cost per pageExact numbers must be right
Human review on low confidenceHigh accuracy on hard pagesCost per pageVolume is high
Trade-off mapInteractive

Good for

Transparent and cheap

Cost

Brittle across layouts

Fails when

Templates change often

  1. Templates change often
  2. Documents are handwritten or scanned poorly
  3. Exact numbers must be right
  4. Volume is high

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: OCR plus rules is good for Transparent and cheap, costs Brittle across layouts, and fails when Templates change often. Layout-aware model is good for Handles varied layouts, costs Needs labelled pages, and fails when Documents are handwritten or scanned poorly. End-to-end vision model is good for No OCR stage to maintain, costs Hallucinated fields and cost per page, and fails when Exact numbers must be right. Human review on low confidence is good for High accuracy on hard pages, costs Cost per page, and fails when Volume is high

Failure modes

  • Reading order collapse. Multi-column pages interleave and extracted sentences become nonsense. Debug: render the extracted text beside the page image and compare column boundaries.
  • Table flattening. Cells shift by one column and numbers attach to the wrong label. Debug: compare parsed table dimensions and headers against the rendered table for a sample.
  • Silent OCR substitution. A digit is misread and passes validation because it is still a number. Debug: cross-check totals against sums of line items and flag mismatches.

In production

Track field-level accuracy on a labelled sample per document type, and the share of pages routed to review. Alert when either moves, since layout changes arrive without notice.

Common misconceptions

People think: Good OCR is enough for documents.

Actually: Text without layout loses which value belongs to which label, which is the actual task.

People think: An end-to-end model removes the need for validation.

Actually: It moves errors from the OCR stage to the generation stage, where they are harder to detect.

Interview angle

Frame the problem as recovering structure rather than reading text, and name reading order and tables as the parts OCR loses. A strong answer describes field-level validation and tracing values to coordinates. Weak answers propose OCR plus a regex for every document type and never discuss layout variation.

Sources