Why it matters
Transformers need a sequence, and an image is a grid of pixels. Splitting it into patches is the conversion, and it fixes two budgets at once: how much detail survives and how many positions the model must attend over. High-resolution documents and screenshots need small patches, while a photo captioning task wastes compute on them. The decision is where on that curve the product sits.
Mental model
Patches are tiles in a mosaic: the tile size decides how much detail survives and how many pieces you must assemble. The metaphor breaks because attention lets every tile see every other tile, so the cost grows with the square of the count rather than linearly.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic document page at 24 × 24 cells, standing in for 16 px patches on a 384 × 384 image, then the same page as a 56 × 28 wide grid for 896 × 448. Patch sizes 1 and 2 cells give 576 and 144 tokens; attention pairs are the exact squares of those counts. The page is a class map, not a legible document.
What to notice. Halving the patch size quadruples the tokens and multiplies attention pairs by sixteen: 20,736 at 144 tokens against 331,776 at 576. Detail and cost move together. A native 896 × 448 grid at 16 px is 56 × 28 = 1,568 tokens, so aspect-ratio handling changes the budget per request, not just the shape. Past the content's bandwidth more patches add cost without legible detail, so pick the resolution at the knee.
Text version
A schematic document with a heading, three paragraph blocks, a four-column table, a logo and a footnote, drawn as a 24 by 24 grid. Steps show 16 px patches (576 tokens), 32 px patches (144 tokens), the row-major mapping token equals row times 12 plus column, the attention-pair comparison 20,736 against 331,776, and the same page as a wide 56 by 28 grid at 1,568 tokens. A slider selects a patch token and reports its row, column and cells.
- Choose patch size from the smallest detail that matters, such as a digit in a table cell or a distant face.
- Compute the resulting token count for your maximum resolution and check it against the model's context.
- Handle aspect ratio explicitly, either by resizing with padding or with a scheme that emits a variable grid.
- For high-resolution inputs, use tiling or dynamic resolution rather than upscaling one global grid.
- Keep preprocessing identical between training and serving, since normalisation and resize interpolation change outputs.
- Measure accuracy against latency at two or three resolutions to find where more patches stop helping.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large patches | Short sequences, fast | Loses small text and fine structure | The task is OCR or detail detection |
| Small patches | Detail for documents and interfaces | Quadratic attention cost | Images are large and latency matters |
| Fixed square grid | Simple and predictable | Distorts aspect ratios | Inputs are wide screenshots or panoramas |
| Dynamic resolution | Native aspect ratios | Variable cost per request | You need predictable latency |
Good for
Short sequences, fast
Cost
Loses small text and fine structure
Fails when
The task is OCR or detail detection
- The task is OCR or detail detection
- Images are large and latency matters
- Inputs are wide screenshots or panoramas
- You need predictable latency
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large patches is good for Short sequences, fast, costs Loses small text and fine structure, and fails when The task is OCR or detail detection. Small patches is good for Detail for documents and interfaces, costs Quadratic attention cost, and fails when Images are large and latency matters. Fixed square grid is good for Simple and predictable, costs Distorts aspect ratios, and fails when Inputs are wide screenshots or panoramas. Dynamic resolution is good for Native aspect ratios, costs Variable cost per request, and fails when You need predictable latency
Failure modes
- Resolution mismatch at serving. Images are resized differently than in training and accuracy drops on fine detail. Debug: log input dimensions and the resize path for live requests and compare with training preprocessing.
- Token overflow. Large images exceed the context and are silently cropped. Debug: assert patch count against the context limit and log whenever truncation occurs.
- Aspect distortion. Wide images are squashed into a square and text becomes unreadable. Debug: inspect the preprocessed tensor for a wide sample and compare character legibility before and after.
In production
Log patch counts per request and cap the maximum resolution, since cost and latency scale with it. Alert when the truncation rate rises, which usually means a new input source.
Common misconceptions
People think: More patches always improve accuracy.
Actually: Past the detail the task needs, extra patches add cost and can dilute attention.
People think: Patch size is a minor implementation detail.
Actually: It sets both the detail ceiling and the compute budget, and changing it invalidates pretrained position handling.
Interview angle
Connect patch size to detail and to sequence length, and say which product requirement fixes it. A strong answer mentions aspect ratio handling and the quadratic cost of attention. Weak answers treat resolution as a free quality dial and never mention that the token count is what actually changes.
Sources
- Dosovitskiy et al., An Image is Worth 16x16 Words — patches as tokens in a vision transformer.
- Dehghani et al., Patch n' Pack: NaViT — variable resolution and sequence packing for images.
- Wang et al., Qwen2-VL — dynamic resolution for documents and varied aspect ratios.