Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 21 of 57

OCR/layout

0 of 57 complete

Why it matters

Scanned contracts, invoices and forms have no text layer, so extraction must recognise characters and decide what the page means structurally. The second job is harder: a form's value is meaningless without its label, and a table's cell is meaningless without its row. OCR engines report per-box confidence, which gives a usable signal for quarantine and for marking low-trust passages at retrieval time.

Mental model

OCR with layout is a human copying a page by hand while drawing boxes around what they see: words plus a sketch of the page. The metaphor breaks because the copyist understands the form, while the pipeline must infer from geometry which box is a label and which is a value.

How it works

A scan as boxes, not paragraphsInteractive

Synthetic scanned claim form · page 1 of 1

No text layer · rendered at 300 dpi

page only

Step 1/6 · Deskew the scan

Step 1Deskew the scan

There is no text layer: the page is an image. Render it at a resolution that keeps the smallest body text legible, then correct skew and noise before recognition.

CLAIMS · SCANNED INTAKE · PAGE 1 OF 1

, running header, region 1. Select to inspect.

Claim intake form

, title, region 2. Select to inspect.
ClaimantM. Okafor
, form field Claimant, region 3. Select to inspect.
PolicyNW-4471
, form field Policy, region 4. Select to inspect.
Date03 / 14 / 26
, form field Date, region 5. Select to inspect.

Incident description

, heading, region 6. Select to inspect.
Text

Windscreen struck by road debris on the northbound carriageway. Driver reported no injuries. Vehicle recovered to the depot the same day.

, paragraph, region 7. Select to inspect.
LINE ITEMS
ItemQtyAmount
Windscreen1480.00
Labour2420.00
Paint1160.00
Total1,060.00
, table LINE ITEMS, region 8. Select to inspect.
PHOTO 1
photo · no text layer
, figure PHOTO 1, region 9. Select to inspect.

Photo 1. Damage reported at intake.

, caption, region 10. Select to inspect.
SignatureM. Okafor
, form field Signature, region 11. Select to inspect.
Received byJ. Adeyemi
, form field Received by, region 12. Select to inspect.
Intake stampRECEIVED · 14 MAR 2026
, form field Intake stamp, region 13. Select to inspect.
Text

Handwritten margin: “second owner”. Not machine-read; keep the page image.

, paragraph, region 14. Select to inspect.

page 1 of 1 · rendered 300 dpi · deskew 0.7°

, running footer, region 15. Select to inspect.
15 regions

Step record

Render
300 dpi
Skew
0.7° corrected
Text layer
none

The text version summarises this figure; select a region for its full text.

Figure. A scanned claim form as OCR sees it: text boxes with coordinates and confidence, not paragraphs. Reading order is reconstructed from geometry, the table is rebuilt from cell boxes, and the two low-confidence regions are what a confidence floor would route to review.

What to notice. Synthetic form. Confidences are authored to demonstrate a threshold, not measured from an OCR engine. The figure shows segmentation, order and routing; it does not claim character accuracy or that one engine behaves this way.

Text version

A synthetic scanned claim form. Form fields hold the claimant, policy number, date, signature and receiving clerk; the policy number is at 71 percent confidence and the stamp at 62 percent, while the other recognised regions are above 90. A left-column incident description sits beside a four-row line-item table, and a hatched photo region with no text layer sits below. The reading-order step numbers the title, fields, paragraph, table, photo, caption, stamp and signatures, with dashed arrows between them. Bounding boxes mark every region, and confidence values appear at the lower right of each. Selecting a region shows its confidence, its state and what the page loses if that region is read out of order.

  1. Render pages at a resolution high enough for the smallest body text, and deskew or denoise before recognition.
  2. Run layout detection to get regions with bounding boxes: title, paragraph, table, figure and footer.
  3. Recognise text per region so language and character settings can differ between a table and prose.
  4. Order regions using column and reading-order heuristics, then join lines within a region.
  5. Reconstruct tables from cell geometry into rows and columns rather than emitting a token stream.
  6. Keep confidence, page number and box coordinates on every block, and quarantine pages below a confidence floor.

Trade-offs

ChoiceGood forCostFails when
Full-page OCR then orderSimple pipelineReading order errors across columnsPages are multi-column or form-like
Layout-first OCRCorrect order and table structureTwo models, more compute per pageVolume is high and pages are simple prose
Low-resolution renderingFast and cheapMisses small type and thin strokesDocuments use small fonts or dense tables
Vision-language extractionHandles handwriting and unusual layoutsCost and output variabilityYou need deterministic, auditable text
Trade-off mapInteractive

Good for

Simple pipeline

Cost

Reading order errors across columns

Fails when

Pages are multi-column or form-like

  1. Pages are multi-column or form-like
  2. Volume is high and pages are simple prose
  3. Documents use small fonts or dense tables
  4. You need deterministic, auditable text

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full-page OCR then order is good for Simple pipeline, costs Reading order errors across columns, and fails when Pages are multi-column or form-like. Layout-first OCR is good for Correct order and table structure, costs Two models, more compute per page, and fails when Volume is high and pages are simple prose. Low-resolution rendering is good for Fast and cheap, costs Misses small type and thin strokes, and fails when Documents use small fonts or dense tables. Vision-language extraction is good for Handles handwriting and unusual layouts, costs Cost and output variability, and fails when You need deterministic, auditable text

Failure modes

  • Reading order inverted across columns. Sentences from the right column follow the left column's lines, producing nonsense that embeds as a distinct topic. Debug: print blocks in output order with their x-coordinates and check the sequence alternates.
  • Table cells detached from headers. Values are indexed without the column they belong to, so a numeric answer is attributed to the wrong field. Debug: inspect the reconstructed table for one known document and verify header association.
  • Silent low-confidence text. A scanned stamp or faint footer is recognised as plausible words and indexed as content. Debug: list blocks below the confidence floor and read a sample.

In production

Track the share of pages below the confidence floor and the OCR error rate on a hand-transcribed sample. Quarantine rather than index low-confidence pages, and keep the page image so a disputed answer can be checked.

Common misconceptions

People think: OCR is a solved problem for printed text.

Actually: Character accuracy can be high while layout reconstruction still fails on columns, forms and tables.

People think: Higher resolution always improves results.

Actually: It helps small text and inflates processing cost and memory, with diminishing returns past a point.

Interview angle

Split recognition from layout reconstruction and argue why order errors hurt more than character errors. Mention confidence as a routing signal and bounding boxes as citation provenance. Treating OCR as one library call, with no account of columns or forms, is the usual gap.

Sources