Why it matters
Hand-built features cannot capture the structure of images, audio or language at scale. Representation learning replaces them with a learned mapping trained on a surrogate objective, so the model discovers which distinctions matter. The engineering consequence is that the embedding becomes an artefact with its own lifecycle: it is versioned, indexed, evaluated on retrieval and reused across tasks that were not part of its training.
Mental model
A representation is a coordinate system chosen so that similar things land near each other and different things land far apart. The metaphor breaks because the axes have no names — you can measure distances but cannot say what any single dimension means.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Six synthetic embeddings in a narrow cone around the diagonal, and one anchor at the low end of it. The axes are two embedding dimensions with no unit and no meaning of their own. Cosine places all six at 1.00 against the anchor, so the top-k is a tie decided in the fourth decimal; euclidean still orders them, but by distance, which here is mostly length. This is a drawing, not output from an encoder, and the whitening that would fix it is not shown.
What to notice. At two decimals every cosine in the panel is 1.00; the largest true difference is under 0.002, and two embeddings are an exact tie because they share a ray. That is what collapse looks like: the index still returns k rows, but their order comes from the third decimal or from the id. Switch to euclidean and the order is distance from the anchor: 0.17, 0.30, 0.48, 0.55, 0.82, 1.04. Every angle here is within a few degrees, so that distance is mostly length — and length is not meaning either. The fix is a different objective, harder negatives, or whitening, not a different metric.
Text version
A scatter of six synthetic embeddings in a narrow cone along the diagonal, with an anchor marked as the query at 0.30, 0.30. The default frame uses cosine and k = 4: all six print 1.00 and the four listed are e2, e5, e1 and e3, with e2 and e5 an exact tie broken by id rather than by any visible geometry. A metric control switches to euclidean, which lists e4 at 0.17, e1 at 0.30, e6 at 0.48 and e2 at 0.55, an order set by distance along the cone. A slider sets k from one to six.
- Choose the objective from the downstream use: contrastive for retrieval, reconstruction for detail preservation, predictive for sequential structure.
- Train or fine-tune the encoder on data that resembles the target domain, since the geometry inherits the training distribution.
- Evaluate with a linear probe or a k-nearest-neighbour check before training a full downstream head.
- Inspect the embedding space for collapse, where all vectors occupy a narrow cone and distances stop discriminating.
- Normalise or whiten the vectors if the downstream index assumes a particular geometry.
- Freeze and version the encoder once it is in use, because re-embedding a corpus is expensive and changes every distance.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Contrastive objective | Strong retrieval geometry | Needs large batches or hard negatives | Positive pairs are noisy |
| Reconstruction objective | Preserves fine detail | May waste capacity on irrelevant variation | The task needs semantic invariance |
| Frozen encoder | Stable index, cheap reuse | Cannot adapt to new domains | The target distribution shifts |
| Fine-tuned encoder | Better domain fit | Re-indexing and evaluation cost | Labels are scarce or the corpus is huge |
Good for
Strong retrieval geometry
Cost
Needs large batches or hard negatives
Fails when
Positive pairs are noisy
- Positive pairs are noisy
- The task needs semantic invariance
- The target distribution shifts
- Labels are scarce or the corpus is huge
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Contrastive objective is good for Strong retrieval geometry, costs Needs large batches or hard negatives, and fails when Positive pairs are noisy. Reconstruction objective is good for Preserves fine detail, costs May waste capacity on irrelevant variation, and fails when The task needs semantic invariance. Frozen encoder is good for Stable index, cheap reuse, costs Cannot adapt to new domains, and fails when The target distribution shifts. Fine-tuned encoder is good for Better domain fit, costs Re-indexing and evaluation cost, and fails when Labels are scarce or the corpus is huge
Failure modes
- Representation collapse. All embeddings become nearly identical and every distance is small. Debug: compute the mean pairwise cosine similarity on a random sample; a value near one indicates collapse.
- Objective mismatch. An encoder trained to cluster topics performs poorly on near-duplicate detection. Debug: evaluate on the actual downstream pair set rather than a generic benchmark.
- Index built on a stale encoder. Half the corpus is embedded with an older model and similarity is meaningless. Debug: store the encoder version beside each vector and count versions in the index.
In production
Monitor embedding norm and pairwise similarity distributions, since collapse and drift both appear there before retrieval metrics move. Keep the encoder and index versions paired, and gate any encoder change with the same retrieval evaluation used for prompts.
Common misconceptions
People think: A better benchmark score means a better representation for your task.
Actually: Representations specialise; a model strong on general similarity can be weak on your domain's distinctions, so the probe must use your data.
People think: Embeddings are interchangeable if the dimension matches.
Actually: Different objectives produce different geometries, so vectors from two encoders are not comparable even at the same dimension.
Interview angle
A strong answer treats the representation as the product, connects the training objective to the geometry it produces, and describes how to evaluate with a probe. It should mention collapse and versioning. Weak answers call embeddings 'features from a neural network' without saying what makes one good or how it is measured.
Sources
- Bengio, Courville & Vincent, Representation Learning: A Review and New Perspectives — why learned representations replace engineered features.
- Mikolov et al., Efficient Estimation of Word Representations in Vector Space — learned embeddings and their geometric structure.
- Radford et al., Learning Transferable Visual Models From Natural Language Supervision — contrastive representation learning and zero-shot transfer.