Why it matters
An image encoder produces vectors; a language model consumes tokens. Bridging them is a translation problem, and the bridge decides how much visual detail survives. Early systems pooled an image into a few vectors, which is cheap and loses spatial information. Later systems project many patch tokens into the language model's context, which preserves detail and costs context length. Instruction tuning then teaches the model what to do with it.
Mental model
A vision-language model is an interpreter standing beside a language model: they see the same scene and translate it into words. The metaphor breaks because the interpreter is trained jointly, so it can omit what it cannot express and the language model will not notice the omission.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic storefront as 576 patch tokens, pooled to 16 vectors or kept per-patch at 144–576 tokens, with an authored cross-attention map for “what does the sign say?”. The scene, weights and answer are illustrations; no model was run.
What to notice. The connector decides what survives: pooling keeps the gist and loses the lettering, per-patch tokens keep the grid and fill the context. Attention can land on the sign without the model reading it. At 64 tokens the lettering spans three patches with no readable glyph, and a fluent “OPEN” answer becomes indistinguishable from a language prior unless a blank-image control is run.
Text version
A schematic storefront with a window, door, awning, a person and a sign with lettering. Steps show 576 patch tokens at 16 px, pooling to 4 by 4 equals 16 tokens, per-patch tokens at 144, an authored cross-attention map whose maximum weight is 1.00 on a sign patch for the question what does the sign say, a downscaled 8 by 8 equals 64-token grid where the sign's lettering spans three patches with no readable glyph, and an answer OPEN marked as not established as grounded. A slider selects patch or pooled tokens and reports coordinates.
- Choose the encoder and its input resolution from the smallest visual detail the task needs.
- Pick the connector: a pooled projection for cheap captioning, or per-patch tokens for reading and counting.
- Align the connector on image-text pairs before instruction tuning, so the language model learns to use the visual tokens.
- Instruction-tune on tasks phrased the way users ask, including refusals when the image does not contain the answer.
- Evaluate on grounded tasks: counting, reading text, spatial relations, and questions about absent objects.
- Check the preprocessing path end to end, since resize and colour handling differ between training and serving.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Pooled visual tokens | Cheap, short context | Loses spatial detail | The task needs reading or counting |
| Per-patch tokens | Detail and grounding | Context cost per image | Many images per request |
| High input resolution | Reads small text | Compute and latency | Throughput matters |
| Instruction-tuned | Follows task phrasing | Needs curated data | The task is pure captioning |
Good for
Cheap, short context
Cost
Loses spatial detail
Fails when
The task needs reading or counting
- The task needs reading or counting
- Many images per request
- Throughput matters
- The task is pure captioning
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Pooled visual tokens is good for Cheap, short context, costs Loses spatial detail, and fails when The task needs reading or counting. Per-patch tokens is good for Detail and grounding, costs Context cost per image, and fails when Many images per request. High input resolution is good for Reads small text, costs Compute and latency, and fails when Throughput matters. Instruction-tuned is good for Follows task phrasing, costs Needs curated data, and fails when The task is pure captioning
Failure modes
- Ungrounded description. The model names objects that are not in the image. Debug: ask about a deliberately absent object and check whether the model asserts its presence.
- Resolution starvation. Text in screenshots is unreadable to the model but legible to a human. Debug: compare answers on the original and downscaled image for the same question.
- Connector undertraining. The model ignores the image and answers from language priors. Debug: evaluate the same questions with a blank image and see whether accuracy barely changes.
In production
Track accuracy on a grounded evaluation set that includes counting and reading, not only caption quality. Cap images per request and resolution, since context cost scales with both.
Common misconceptions
People think: A VLM sees the image the way a person does.
Actually: It sees a resized grid of patches through an encoder, with a fixed detail ceiling.
People think: Better language fluency means better visual grounding.
Actually: Fluent answers can be produced from priors without using the image at all.
Interview angle
Explain the connector as the place where visual detail is either preserved or lost, and say how that choice shows up in tasks. A strong answer proposes the blank-image test to detect a model ignoring its input. Weak answers treat the encoder as the whole system and evaluate only caption fluency.
Sources
- Liu et al., Visual Instruction Tuning — LLaVA and instruction tuning over visual tokens.
- Alayrac et al., Flamingo: a Visual Language Model for Few-Shot Learning — interleaving visual features into a language model.
- Li et al., BLIP-2 — a lightweight querying connector between frozen encoders.