Why it matters
Every downstream number in a RAG system — similarity floor, top-k, rerank depth — is calibrated against one embedding space. The model therefore behaves like a schema: changing it invalidates stored vectors, cached scores and tuned thresholds at once. Teams get into trouble by choosing on a public average, then discovering after indexing that the model truncates their long clauses, ignores their language or collapses product codes into one region of the space.
Mental model
An embedding model is a lossy compression codec for meaning, with a fixed bit budget per document. The codec is stable and fast, and it is also irreversible: you cannot recover the sentence that the compression dropped, only observe that retrieval stopped finding it.
How it works
- 1Measure the token-length distribution of your chunks and reject any model whose maximum input is below the long tail.
- 2Check the model card for the required pooling, normalisation and query or passage instruction prefixes, then apply them identically at index and query time.
- 3Evaluate candidates on a labelled slice of real queries, reporting recall@k per query type rather than one average.
- 4Compare dimension against memory and latency budget: a smaller vector reduces index size and query cost proportionally.
- 5Confirm language and domain coverage with sample queries in each language and with identifier-heavy queries.
- 6Record model name, version, dimension, normalisation and prefix convention in index metadata before the first ingest.
Step 1 of 6
Measure the token-length distribution of your chunks and reject any model whose maximum input is below the long tail.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Measure the token-length distribution of your chunks and reject any model whose maximum i… 2. Check the model card for the required pooling, normalisation and query or passage instruc… 3. Evaluate candidates on a labelled slice of real queries, reporting recall@k per query typ… 4. Compare dimension against memory and latency budget: a smaller vector reduces index size … 5. Confirm language and domain coverage with sample queries in each language and with identi… 6. Record model name, version, dimension, normalisation and prefix convention in index metad…
- Measure the token-length distribution of your chunks and reject any model whose maximum input is below the long tail.
- Check the model card for the required pooling, normalisation and query or passage instruction prefixes, then apply them identically at index and query time.
- Evaluate candidates on a labelled slice of real queries, reporting recall@k per query type rather than one average.
- Compare dimension against memory and latency budget: a smaller vector reduces index size and query cost proportionally.
- Confirm language and domain coverage with sample queries in each language and with identifier-heavy queries.
- Record model name, version, dimension, normalisation and prefix convention in index metadata before the first ingest.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large general model | Broad coverage and strong paraphrase recall | Memory, latency and per-token cost | The corpus is large and the latency budget is tight |
| Small distilled model | Cheap to serve and index | Weaker on rare vocabulary and long inputs | Queries are subtle or the domain is specialised |
| Long-context encoder | Fewer chunks and less context splitting | Cost per document and diluted vectors | Answers are single facts inside long documents |
| Domain-tuned model | Precision on internal jargon and codes | Labelling effort and a re-index per version | Traffic mixes many unrelated domains |
Good for
Broad coverage and strong paraphrase recall
Cost
Memory, latency and per-token cost
Fails when
The corpus is large and the latency budget is tight
- The corpus is large and the latency budget is tight
- Queries are subtle or the domain is specialised
- Answers are single facts inside long documents
- Traffic mixes many unrelated domains
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large general model is good for Broad coverage and strong paraphrase recall, costs Memory, latency and per-token cost, and fails when The corpus is large and the latency budget is tight. Small distilled model is good for Cheap to serve and index, costs Weaker on rare vocabulary and long inputs, and fails when Queries are subtle or the domain is specialised. Long-context encoder is good for Fewer chunks and less context splitting, costs Cost per document and diluted vectors, and fails when Answers are single facts inside long documents. Domain-tuned model is good for Precision on internal jargon and codes, costs Labelling effort and a re-index per version, and fails when Traffic mixes many unrelated domains
Failure modes
- Instruction prefix omitted at query time. Queries are embedded in passage mode, so the query sits in the wrong region and recall drops quietly. Debug: embed one labelled query both ways and compare its rank against the known answer.
- Truncation of long chunks. The decisive clause is cut before encoding, so the chunk's vector describes only its opening. Debug: count tokens per chunk and list those above the model maximum.
- Mixed model versions in one index. Distances between vectors from two models are meaningless, yet search returns results without error. Debug: group stored vectors by model identifier and confirm only one version is present.
In production
Version the embedding model with the index and gate upgrades behind the labelled recall set. Keep the previous index queryable until the new one passes, and alert on any request that embeds with a different model than the index was built with.
Common misconceptions
People think: A larger dimension always retrieves better.
Actually: Dimension adds representational capacity but also memory and latency; gains on your corpus must be measured.
People think: One model can serve every language and domain.
Actually: Coverage is uneven, and a model strong on English prose can be weak on code, tables or low-resource languages.
Interview angle
Frame the model as index schema: the preprocessing contract, the migration plan and the evaluation that gates an upgrade. Discuss truncation and dimension as concrete constraints rather than preferences. Answers that name a model and a leaderboard score, without saying what would force a full re-index, miss the operational core of the decision.
Sources
- Muennighoff et al., MTEB — why embedding quality must be read per task rather than as one average.
- Reimers & Gurevych, Sentence-BERT — pooling strategies and the encoder design behind sentence embeddings.
- OpenAI, Embeddings guide — input limits, dimensionality and normalisation in a hosted embedding API.