Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 3 of 22

VLMs

0 of 22 complete

Why it matters

An image encoder produces vectors; a language model consumes tokens. Bridging them is a translation problem, and the bridge decides how much visual detail survives. Early systems pooled an image into a few vectors, which is cheap and loses spatial information. Later systems project many patch tokens into the language model's context, which preserves detail and costs context length. Instruction tuning then teaches the model what to do with it.

Mental model

A vision-language model is an interpreter standing beside a language model: they see the same scene and translate it into words. The metaphor breaks because the interpreter is trained jointly, so it can omit what it cannot express and the language model will not notice the omission.

How it works

Patches, a connector, and where attention landsInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic storefront as 576 patch tokens, pooled to 16 vectors or kept per-patch at 144–576 tokens, with an authored cross-attention map for “what does the sign say?”. The scene, weights and answer are illustrations; no model was run.

What to notice. The connector decides what survives: pooling keeps the gist and loses the lettering, per-patch tokens keep the grid and fill the context. Attention can land on the sign without the model reading it. At 64 tokens the lettering spans three patches with no readable glyph, and a fluent “OPEN” answer becomes indistinguishable from a language prior unless a blank-image control is run.

Text version

A schematic storefront with a window, door, awning, a person and a sign with lettering. Steps show 576 patch tokens at 16 px, pooling to 4 by 4 equals 16 tokens, per-patch tokens at 144, an authored cross-attention map whose maximum weight is 1.00 on a sign patch for the question what does the sign say, a downscaled 8 by 8 equals 64-token grid where the sign's lettering spans three patches with no readable glyph, and an answer OPEN marked as not established as grounded. A slider selects patch or pooled tokens and reports coordinates.

  1. Choose the encoder and its input resolution from the smallest visual detail the task needs.
  2. Pick the connector: a pooled projection for cheap captioning, or per-patch tokens for reading and counting.
  3. Align the connector on image-text pairs before instruction tuning, so the language model learns to use the visual tokens.
  4. Instruction-tune on tasks phrased the way users ask, including refusals when the image does not contain the answer.
  5. Evaluate on grounded tasks: counting, reading text, spatial relations, and questions about absent objects.
  6. Check the preprocessing path end to end, since resize and colour handling differ between training and serving.

Trade-offs

ChoiceGood forCostFails when
Pooled visual tokensCheap, short contextLoses spatial detailThe task needs reading or counting
Per-patch tokensDetail and groundingContext cost per imageMany images per request
High input resolutionReads small textCompute and latencyThroughput matters
Instruction-tunedFollows task phrasingNeeds curated dataThe task is pure captioning
Trade-off mapInteractive

Good for

Cheap, short context

Cost

Loses spatial detail

Fails when

The task needs reading or counting

  1. The task needs reading or counting
  2. Many images per request
  3. Throughput matters
  4. The task is pure captioning

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Pooled visual tokens is good for Cheap, short context, costs Loses spatial detail, and fails when The task needs reading or counting. Per-patch tokens is good for Detail and grounding, costs Context cost per image, and fails when Many images per request. High input resolution is good for Reads small text, costs Compute and latency, and fails when Throughput matters. Instruction-tuned is good for Follows task phrasing, costs Needs curated data, and fails when The task is pure captioning

Failure modes

  • Ungrounded description. The model names objects that are not in the image. Debug: ask about a deliberately absent object and check whether the model asserts its presence.
  • Resolution starvation. Text in screenshots is unreadable to the model but legible to a human. Debug: compare answers on the original and downscaled image for the same question.
  • Connector undertraining. The model ignores the image and answers from language priors. Debug: evaluate the same questions with a blank image and see whether accuracy barely changes.

In production

Track accuracy on a grounded evaluation set that includes counting and reading, not only caption quality. Cap images per request and resolution, since context cost scales with both.

Common misconceptions

People think: A VLM sees the image the way a person does.

Actually: It sees a resized grid of patches through an encoder, with a fixed detail ceiling.

People think: Better language fluency means better visual grounding.

Actually: Fluent answers can be produced from priors without using the image at all.

Interview angle

Explain the connector as the place where visual detail is either preserved or lost, and say how that choice shows up in tasks. A strong answer proposes the blank-image test to detect a model ignoring its input. Weak answers treat the encoder as the whole system and evaluate only caption fluency.

Sources