Why it matters
Scanned contracts, invoices and forms have no text layer, so extraction must recognise characters and decide what the page means structurally. The second job is harder: a form's value is meaningless without its label, and a table's cell is meaningless without its row. OCR engines report per-box confidence, which gives a usable signal for quarantine and for marking low-trust passages at retrieval time.
Mental model
OCR with layout is a human copying a page by hand while drawing boxes around what they see: words plus a sketch of the page. The metaphor breaks because the copyist understands the form, while the pipeline must infer from geometry which box is a label and which is a value.
How it works
Synthetic scanned claim form · page 1 of 1
No text layer · rendered at 300 dpi
page only
Step 1Deskew the scan
There is no text layer: the page is an image. Render it at a resolution that keeps the smallest body text legible, then correct skew and noise before recognition.
CLAIMS · SCANNED INTAKE · PAGE 1 OF 1
Claim intake form
Incident description
| Item | Qty | Amount |
|---|---|---|
| Windscreen | 1 | 480.00 |
| Labour | 2 | 420.00 |
| Paint | 1 | 160.00 |
| Total | 1,060.00 |
Photo 1. Damage reported at intake.
page 1 of 1 · rendered 300 dpi · deskew 0.7°
Step record
- Render
- 300 dpi
- Skew
- 0.7° corrected
- Text layer
- none
The text version summarises this figure; select a region for its full text.
Figure. A scanned claim form as OCR sees it: text boxes with coordinates and confidence, not paragraphs. Reading order is reconstructed from geometry, the table is rebuilt from cell boxes, and the two low-confidence regions are what a confidence floor would route to review.
What to notice. Synthetic form. Confidences are authored to demonstrate a threshold, not measured from an OCR engine. The figure shows segmentation, order and routing; it does not claim character accuracy or that one engine behaves this way.
Text version
A synthetic scanned claim form. Form fields hold the claimant, policy number, date, signature and receiving clerk; the policy number is at 71 percent confidence and the stamp at 62 percent, while the other recognised regions are above 90. A left-column incident description sits beside a four-row line-item table, and a hatched photo region with no text layer sits below. The reading-order step numbers the title, fields, paragraph, table, photo, caption, stamp and signatures, with dashed arrows between them. Bounding boxes mark every region, and confidence values appear at the lower right of each. Selecting a region shows its confidence, its state and what the page loses if that region is read out of order.
- Render pages at a resolution high enough for the smallest body text, and deskew or denoise before recognition.
- Run layout detection to get regions with bounding boxes: title, paragraph, table, figure and footer.
- Recognise text per region so language and character settings can differ between a table and prose.
- Order regions using column and reading-order heuristics, then join lines within a region.
- Reconstruct tables from cell geometry into rows and columns rather than emitting a token stream.
- Keep confidence, page number and box coordinates on every block, and quarantine pages below a confidence floor.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full-page OCR then order | Simple pipeline | Reading order errors across columns | Pages are multi-column or form-like |
| Layout-first OCR | Correct order and table structure | Two models, more compute per page | Volume is high and pages are simple prose |
| Low-resolution rendering | Fast and cheap | Misses small type and thin strokes | Documents use small fonts or dense tables |
| Vision-language extraction | Handles handwriting and unusual layouts | Cost and output variability | You need deterministic, auditable text |
Good for
Simple pipeline
Cost
Reading order errors across columns
Fails when
Pages are multi-column or form-like
- Pages are multi-column or form-like
- Volume is high and pages are simple prose
- Documents use small fonts or dense tables
- You need deterministic, auditable text
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full-page OCR then order is good for Simple pipeline, costs Reading order errors across columns, and fails when Pages are multi-column or form-like. Layout-first OCR is good for Correct order and table structure, costs Two models, more compute per page, and fails when Volume is high and pages are simple prose. Low-resolution rendering is good for Fast and cheap, costs Misses small type and thin strokes, and fails when Documents use small fonts or dense tables. Vision-language extraction is good for Handles handwriting and unusual layouts, costs Cost and output variability, and fails when You need deterministic, auditable text
Failure modes
- Reading order inverted across columns. Sentences from the right column follow the left column's lines, producing nonsense that embeds as a distinct topic. Debug: print blocks in output order with their x-coordinates and check the sequence alternates.
- Table cells detached from headers. Values are indexed without the column they belong to, so a numeric answer is attributed to the wrong field. Debug: inspect the reconstructed table for one known document and verify header association.
- Silent low-confidence text. A scanned stamp or faint footer is recognised as plausible words and indexed as content. Debug: list blocks below the confidence floor and read a sample.
In production
Track the share of pages below the confidence floor and the OCR error rate on a hand-transcribed sample. Quarantine rather than index low-confidence pages, and keep the page image so a disputed answer can be checked.
Common misconceptions
People think: OCR is a solved problem for printed text.
Actually: Character accuracy can be high while layout reconstruction still fails on columns, forms and tables.
People think: Higher resolution always improves results.
Actually: It helps small text and inflates processing cost and memory, with diminishing returns past a point.
Interview angle
Split recognition from layout reconstruction and argue why order errors hurt more than character errors. Mention confidence as a routing signal and bounding boxes as citation provenance. Treating OCR as one library call, with no account of columns or forms, is the usual gap.
Sources
- Tesseract OCR — open-source recognition, page segmentation modes and confidence output.
- Huang et al., LayoutLMv3 — joint text and layout modelling for document understanding.
- PyMuPDF documentation — rendering pages and reading block geometry for layout work.