Why it matters
A PDF is a rendering, not a data structure. The same text appears in a header, a footnote and a table cell, and only position distinguishes them. Pipelines that flatten a page to a string lose the relationships that make an invoice an invoice. Document understanding exists to recover those relationships, and its failures are expensive because they surface as wrong numbers downstream.
Mental model
Document understanding is rebuilding a form from a photograph: you need the boxes and their positions, not only the words inside them. The metaphor breaks because a model can learn layout conventions, so it can infer a field's role from position even when the label above it is ambiguous.
How it works
Synthetic supplier invoice · page 1 of 1
Five fields, one line-item table, one validation rule
bounding boxes
Step 1Regions before fields
The invoice number sits top-right, the vendor top-left, line items in the middle and the total at the bottom right. Position is part of the label: the same string can appear in several roles.
NORTHWIND SUPPLIES · INVOICE · PAGE 1 OF 1
Invoice NW-8841
| Description | Qty | Unit | Amount |
|---|---|---|---|
| Valve assembly | 12 | 640.00 | 7,680.00 |
| Seal kit | 40 | 48.00 | !1,820.00 |
| Freight | 1 | 380.00 | 380.00 |
| Subtotal | 9,880.00 | ||
| Tax | 2,500.00 | ||
| Total | ✕12,480.00 |
extracted 2026-11-17 09:41 · 5 fields · 1 line item low · 1 page routed
Step record
- Document type
- invoice
- Target
- fields + table
- Page
- 1 of 1
The text version summarises this figure; select a region for its full text.
Figure. An invoice page as a field-extraction target: vendor, dates, line items and total are located by layout, read with confidence and validated by arithmetic. One misread line-item amount makes the total check fail, which routes the page to review instead of accepting a plausible number.
What to notice. Synthetic invoice. All values, coordinates and confidences are authored; the arithmetic is exact for the numbers shown. The figure demonstrates a validation rule, not a model's measured field accuracy.
Text version
A synthetic supplier invoice. Fields hold the invoice number, date, vendor, purchase order and due date. A six-row line-item table lists valve assemblies, seal kits, freight, subtotal, tax and total. The seal-kit amount reads 1,820.00 at 78 percent confidence; the line items sum to 9,880.00 and with 2,500.00 tax give 12,380.00 against a printed total of 12,480.00, so the total row is marked failed. A validation block shows the mismatch, a review block records the field and the 100.00 difference, and an extraction record gives the field count, the low cell and the review route. Steps show bounding boxes, confidence, the rebuilt table, the arithmetic failure, the review route and the per-field evaluation.
- Classify the document types in scope, since invoices, contracts and scientific papers need different schemas.
- Decide the extraction target: plain text with reading order, key-value fields, or full table structure.
- Choose the pipeline shape, either OCR plus a layout model or an end-to-end model that reads page images.
- Preserve coordinates through the pipeline so every extracted value can be traced to a region on the page.
- Validate extracted fields with type and range checks, and route failures to review rather than guessing.
- Evaluate per field and per document type, because an average over pages hides the field that is always wrong.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| OCR plus rules | Transparent and cheap | Brittle across layouts | Templates change often |
| Layout-aware model | Handles varied layouts | Needs labelled pages | Documents are handwritten or scanned poorly |
| End-to-end vision model | No OCR stage to maintain | Hallucinated fields and cost per page | Exact numbers must be right |
| Human review on low confidence | High accuracy on hard pages | Cost per page | Volume is high |
Good for
Transparent and cheap
Cost
Brittle across layouts
Fails when
Templates change often
- Templates change often
- Documents are handwritten or scanned poorly
- Exact numbers must be right
- Volume is high
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: OCR plus rules is good for Transparent and cheap, costs Brittle across layouts, and fails when Templates change often. Layout-aware model is good for Handles varied layouts, costs Needs labelled pages, and fails when Documents are handwritten or scanned poorly. End-to-end vision model is good for No OCR stage to maintain, costs Hallucinated fields and cost per page, and fails when Exact numbers must be right. Human review on low confidence is good for High accuracy on hard pages, costs Cost per page, and fails when Volume is high
Failure modes
- Reading order collapse. Multi-column pages interleave and extracted sentences become nonsense. Debug: render the extracted text beside the page image and compare column boundaries.
- Table flattening. Cells shift by one column and numbers attach to the wrong label. Debug: compare parsed table dimensions and headers against the rendered table for a sample.
- Silent OCR substitution. A digit is misread and passes validation because it is still a number. Debug: cross-check totals against sums of line items and flag mismatches.
In production
Track field-level accuracy on a labelled sample per document type, and the share of pages routed to review. Alert when either moves, since layout changes arrive without notice.
Common misconceptions
People think: Good OCR is enough for documents.
Actually: Text without layout loses which value belongs to which label, which is the actual task.
People think: An end-to-end model removes the need for validation.
Actually: It moves errors from the OCR stage to the generation stage, where they are harder to detect.
Interview angle
Frame the problem as recovering structure rather than reading text, and name reading order and tables as the parts OCR loses. A strong answer describes field-level validation and tracing values to coordinates. Weak answers propose OCR plus a regex for every document type and never discuss layout variation.
Sources
- Kim et al., Donut: Document Understanding Transformer without OCR — end-to-end page-image parsing without a separate OCR stage.
- Huang et al., LayoutLMv3 — joint text, layout and image modelling for documents.
- Mathew et al., DocVQA — question answering over document images as an evaluation task.