Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 24 of 27

Representation learning

0 of 27 complete

Why it matters

Hand-built features cannot capture the structure of images, audio or language at scale. Representation learning replaces them with a learned mapping trained on a surrogate objective, so the model discovers which distinctions matter. The engineering consequence is that the embedding becomes an artefact with its own lifecycle: it is versioned, indexed, evaluated on retrieval and reused across tasks that were not part of its training.

Mental model

A representation is a coordinate system chosen so that similar things land near each other and different things land far apart. The metaphor breaks because the axes have no names — you can measure distances but cannot say what any single dimension means.

How it works

Collapse: a tie at two decimalsInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Six synthetic embeddings in a narrow cone around the diagonal, and one anchor at the low end of it. The axes are two embedding dimensions with no unit and no meaning of their own. Cosine places all six at 1.00 against the anchor, so the top-k is a tie decided in the fourth decimal; euclidean still orders them, but by distance, which here is mostly length. This is a drawing, not output from an encoder, and the whitening that would fix it is not shown.

What to notice. At two decimals every cosine in the panel is 1.00; the largest true difference is under 0.002, and two embeddings are an exact tie because they share a ray. That is what collapse looks like: the index still returns k rows, but their order comes from the third decimal or from the id. Switch to euclidean and the order is distance from the anchor: 0.17, 0.30, 0.48, 0.55, 0.82, 1.04. Every angle here is within a few degrees, so that distance is mostly length — and length is not meaning either. The fix is a different objective, harder negatives, or whitening, not a different metric.

Text version

A scatter of six synthetic embeddings in a narrow cone along the diagonal, with an anchor marked as the query at 0.30, 0.30. The default frame uses cosine and k = 4: all six print 1.00 and the four listed are e2, e5, e1 and e3, with e2 and e5 an exact tie broken by id rather than by any visible geometry. A metric control switches to euclidean, which lists e4 at 0.17, e1 at 0.30, e6 at 0.48 and e2 at 0.55, an order set by distance along the cone. A slider sets k from one to six.

  1. Choose the objective from the downstream use: contrastive for retrieval, reconstruction for detail preservation, predictive for sequential structure.
  2. Train or fine-tune the encoder on data that resembles the target domain, since the geometry inherits the training distribution.
  3. Evaluate with a linear probe or a k-nearest-neighbour check before training a full downstream head.
  4. Inspect the embedding space for collapse, where all vectors occupy a narrow cone and distances stop discriminating.
  5. Normalise or whiten the vectors if the downstream index assumes a particular geometry.
  6. Freeze and version the encoder once it is in use, because re-embedding a corpus is expensive and changes every distance.

Trade-offs

ChoiceGood forCostFails when
Contrastive objectiveStrong retrieval geometryNeeds large batches or hard negativesPositive pairs are noisy
Reconstruction objectivePreserves fine detailMay waste capacity on irrelevant variationThe task needs semantic invariance
Frozen encoderStable index, cheap reuseCannot adapt to new domainsThe target distribution shifts
Fine-tuned encoderBetter domain fitRe-indexing and evaluation costLabels are scarce or the corpus is huge
Trade-off mapInteractive

Good for

Strong retrieval geometry

Cost

Needs large batches or hard negatives

Fails when

Positive pairs are noisy

  1. Positive pairs are noisy
  2. The task needs semantic invariance
  3. The target distribution shifts
  4. Labels are scarce or the corpus is huge

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Contrastive objective is good for Strong retrieval geometry, costs Needs large batches or hard negatives, and fails when Positive pairs are noisy. Reconstruction objective is good for Preserves fine detail, costs May waste capacity on irrelevant variation, and fails when The task needs semantic invariance. Frozen encoder is good for Stable index, cheap reuse, costs Cannot adapt to new domains, and fails when The target distribution shifts. Fine-tuned encoder is good for Better domain fit, costs Re-indexing and evaluation cost, and fails when Labels are scarce or the corpus is huge

Failure modes

  • Representation collapse. All embeddings become nearly identical and every distance is small. Debug: compute the mean pairwise cosine similarity on a random sample; a value near one indicates collapse.
  • Objective mismatch. An encoder trained to cluster topics performs poorly on near-duplicate detection. Debug: evaluate on the actual downstream pair set rather than a generic benchmark.
  • Index built on a stale encoder. Half the corpus is embedded with an older model and similarity is meaningless. Debug: store the encoder version beside each vector and count versions in the index.

In production

Monitor embedding norm and pairwise similarity distributions, since collapse and drift both appear there before retrieval metrics move. Keep the encoder and index versions paired, and gate any encoder change with the same retrieval evaluation used for prompts.

Common misconceptions

People think: A better benchmark score means a better representation for your task.

Actually: Representations specialise; a model strong on general similarity can be weak on your domain's distinctions, so the probe must use your data.

People think: Embeddings are interchangeable if the dimension matches.

Actually: Different objectives produce different geometries, so vectors from two encoders are not comparable even at the same dimension.

Interview angle

A strong answer treats the representation as the product, connects the training objective to the geometry it produces, and describes how to evaluate with a probe. It should mention collapse and versioning. Weak answers call embeddings 'features from a neural network' without saying what makes one good or how it is measured.

Sources