Why it matters
A model cannot multiply text. Before anything useful happens, a token, an image patch, an audio frame or a user profile is converted into a fixed-length list of numbers. That list is a vector, and every downstream operation — attention, retrieval, classification, clustering — is arithmetic on those lists.
This is why vector questions keep appearing in engineering work that looks unrelated. Embedding storage is a memory problem measured in gigabytes. Retrieval quality is a geometry problem. A model upgrade that changes the embedding space is a data migration. If you cannot reason about the shape, dtype and normalisation of a vector, you cannot predict what a vector system will cost or how it will fail.
Mental model: coordinates in a space you cannot draw
A vector is a set of coordinates. A two-dimensional vector [3, 4] names a point three units east and four units north. A 768-dimensional vector names a point in a space with 768 axes — impossible to picture, but the same arithmetic applies.
The analogy is useful because it turns algebra into movement: adding vectors moves you, scaling a vector stretches the arrow, and the dot product measures how much two arrows point the same way.
It breaks in one important place. In ordinary geometry the axes have names and meaning: east, north, up. In a learned embedding space the axes are arbitrary. Coordinate 412 does not mean "is a refund policy". Only the relative geometry — distances, angles, neighbourhoods — carries meaning, and that meaning is whatever the training objective forced into it. Treat a vector as a coordinate, not as a list of interpretable features.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Five synthetic vectors built from a = (0.50, 0.25) and b = (−0.10, 0.50), plotted by their two components. The query is a. Cosine measures the angle from a; euclidean measures the distance to it. Scaling a keeps the direction, addition and subtraction change it, and the zero vector has no direction at all. The axes are the two components, no unit. Three dimensions and the arithmetic that produced the sums are not drawn.
What to notice. 2a is twice as long and scores cosine 1.00 with a, and the distance from a to 2a is 0.56 — exactly a's length. Adding b moves off the ray: a + b scores 0.82 and a − b scores 0.65, so the two sums sit on opposite sides of a and neither is a scaled copy of it. The zero vector sits at the origin and scores 0.00, because it has no direction to compare. Euclidean repeats the same lengths from the other side: 0.56 to 2a and to the origin, and 0.51 to either sum, which is |b|.
Text version
A scatter of five synthetic vectors, with a = (0.50, 0.25) marked as the query. The default frame uses cosine and k = 5: 2a scores 1.00, a + b 0.82, a − b 0.65, b 0.26 and the zero vector 0.00, all joined to a and listed in that order. 2a lies on a's ray at twice the length, while a + b and a − b sit on opposite sides of the ray. A metric control switches to euclidean, which lists a + b and a − b at 0.51, the origin and 2a at 0.56, and b at 0.65.
- Fix the representation. An item becomes . The length is the dimension. For an embedding table, is a hyperparameter chosen at training time; for a transformer's hidden state, is the model width.
- Index by position, not by name. Position carries meaning, so
[1, 2]and[2, 1]are different vectors. This ordering is why shape bugs are so common: a transposed matrix still contains the right numbers. - Combine with two operations. Vector addition produces a new point (used for combining representations, and for the residual stream in transformers). Scalar multiplication stretches or shrinks a vector without changing its direction.
- Measure length. The Euclidean norm is the distance from the origin. Dividing by it produces a unit vector, which is the usual precondition for cosine similarity.
- Turn discrete things into vectors. A token id indexes a row of an embedding matrix , where is the vocabulary size. Retrieval systems do the reverse: a whole document is encoded by one vector produced by an encoder model.
- Choose a dtype and pay for it. float32 uses 4 bytes per component, float16 and bfloat16 use 2, int8 uses 1. Memory is times bytes per component per vector; a million 768-dimensional float32 vectors occupy about 3 GB before index structures.
- Compare with a fixed metric. Dot product, cosine similarity and Euclidean distance are all defined on the same objects and can disagree about ranking when magnitudes differ.
- Shape
- one vector: (d,); a batch: (batch, d)
- Memory
- d × bytes per component, per vector
- float32 size
- 4 bytes per dimension
- Unit vector
- norm = 1, so dot product equals cosine
The arithmetic that matters
Two operations cover most of the work. The dot product measures alignment, and the norm measures size:
Cosine similarity divides the dot product by both norms so that only direction counts:
Storage follows directly from the definitions:
A minimal example
import numpy as np
vocab_size, d = 50_000, 768
rng = np.random.default_rng(0)
E = rng.normal(0, 0.02, size=(vocab_size, d)).astype(np.float32) # (V, d)
def embed(ids: list[int]) -> np.ndarray:
return E[np.asarray(ids)] # (len(ids), d)
def unit(v: np.ndarray) -> np.ndarray:
norms = np.linalg.norm(v, axis=-1, keepdims=True)
return v / np.maximum(norms, 1e-12) # zero vectors stay zero
x = unit(embed([101, 202])) # (2, d)
y = unit(embed([303])) # (1, d)
similarity = x @ y.T # (2, 1) cosine scores
print(x.shape, y.shape, similarity.shape) # (2, 768) (1, 768) (2, 1)
print(f"embedding table: {E.nbytes / 1e6:.1f} MB")The keepdims=True is not cosmetic: without it the norm has shape (2,), broadcasting silently divides the wrong axis and every downstream score is wrong. Print shapes at every boundary while learning this.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| float32 vectors | Exact ranking, small corpora | 4 bytes per dimension | Memory budget cannot hold the index |
| float16 / bfloat16 | Large indexes on GPU | Small ranking drift, needs calibration | No ground truth to measure the drift against |
| int8 quantisation | Billions of vectors in RAM | Needs rescoring to recover precision | Quantised inner products replace the final ranking |
| High dimension (1024+) | Fine-grained distinctions | Memory and latency grow with d | Corpus is small and distances saturate |
| Low dimension (128–256) | Cheap, fast, edge deployment | Loses fine detail | Near-duplicates must be separated |
| Dense vector only | Semantic paraphrase | Poor on identifiers and codes | Query contains an order number or SKU |
Good for
Exact ranking, small corpora
Cost
4 bytes per dimension
Fails when
Memory budget cannot hold the index
- Memory budget cannot hold the index
- No ground truth to measure the drift against
- Quantised inner products replace the final ranking
- Corpus is small and distances saturate
- Near-duplicates must be separated
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: float32 vectors is good for Exact ranking, small corpora, costs 4 bytes per dimension, and fails when Memory budget cannot hold the index. float16 / bfloat16 is good for Large indexes on GPU, costs Small ranking drift, needs calibration, and fails when No ground truth to measure the drift against. int8 quantisation is good for Billions of vectors in RAM, costs Needs rescoring to recover precision, and fails when Quantised inner products replace the final ranking. High dimension (1024+) is good for Fine-grained distinctions, costs Memory and latency grow with d, and fails when Corpus is small and distances saturate. Low dimension (128–256) is good for Cheap, fast, edge deployment, costs Loses fine detail, and fails when Near-duplicates must be separated
Failure modes
Axis confusion. A batch assembled as (d, batch) instead of (batch, d) produces scores of plausible magnitude that are pure noise. Debugging step: assert the last axis equals the known dimension at every function boundary and print shapes on the first request.
Mixed normalisation. If the index stores unit vectors and the query is not normalised, a dot-product index no longer computes cosine similarity, and rankings shift quietly. Debugging step: log the distribution of vector norms in the index and at query time; they should match.
Embedding version skew. The query encoder is upgraded but the corpus is not re-embedded, so queries and documents live in different spaces. Debugging step: store the encoder name, version and dimension in index metadata, and refuse to serve when the query encoder's identifiers differ.
Quantisation without rescoring. Compressing vectors to int8 shrinks memory, but the compressed inner product is an approximation. Debugging step: sample a few hundred queries and compare recall against exact float32 search.
In production
Vector memory is usually the first hard limit you hit. Compute it before choosing an index: vectors × dimension × bytes, plus index overhead and replicas. Then decide whether the dtype, the dimension, or the corpus is the thing to reduce.
Normalise at one layer only. Pick either "store unit vectors and use inner product" or "store raw vectors and use a cosine metric", document it, and enforce it in the encoder client rather than in each caller. Most silent retrieval regressions come from this rule being broken once.
Treat an embedding-model change as a migration, not a deployment. Build the new index alongside the old one, measure recall of the new index against labelled pairs, and cut over with a rollback path. The vectors are derived data, but regenerating them is a full corpus pass.
Common misconceptions
People think: Individual numbers in an embedding have meanings you can read off.
Actually: Axes in a learned space are arbitrary. Only distances and directions between vectors are meaningful, and only relative to the data the encoder was trained on.
People think: More dimensions always means better retrieval.
Actually: Higher dimension can separate finer distinctions, but it costs memory and latency linearly and saturates on small corpora. Dimension is a trade-off, not a quality dial.
A third confusion is subtler: that a vector is understanding. It is a compressed summary produced by an objective. If the objective never rewarded distinguishing two similar products, no amount of vector arithmetic will separate them.
Interview angle
Vector questions test whether you can move between the geometric picture and the operational bill. Expect: why cosine rather than Euclidean, what a dimension buys you, how much memory ten million chunks need, and what happens to a live index when the embedding model changes. Strong answers name the units and do the arithmetic out loud. See cosine similarity, embedding models, and the retrieval questions under /interview/rag.
Sources
- Mikolov et al., Efficient Estimation of Word Representations in Vector Space — the word2vec formulation that made learned vectors standard: https://arxiv.org/abs/1301.3781
- Jay Alammar, The Illustrated Word2vec — visual explanation of embedding tables and vector arithmetic: https://jalammar.github.io/illustrated-word2vec/
- Johnson, Douze & Jégou, Billion-scale Similarity Search with GPUs — memory and quantisation constraints at index scale: https://arxiv.org/abs/1702.08734
- 3Blue1Brown, Essence of Linear Algebra — the geometric reading of vectors, spans and transformations: https://www.3blue1brown.com/topics/linear-algebra