Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 22 of 27

Clustering & dimensionality reduction

0 of 27 complete

Why it matters

Exploration is where labels do not exist yet: you want to know whether the data has natural groups, which examples are anomalous, or whether a hundred features collapse into a few directions. Clustering and dimensionality reduction make those questions tractable, but they always return an answer — even for structureless data. The engineering problem is that the output is a hypothesis, and treating it as a finding is how spurious segments reach product decisions.

Mental model

Clustering is sorting a pile of photos into stacks by resemblance; dimensionality reduction is describing each photo with two numbers instead of a hundred. The metaphor breaks because you choose the number of stacks and the number of numbers in advance, and that choice shapes what you find.

How it works

The assignment depends on the distanceInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Three synthetic cluster centroids and one unassigned point in a hand-placed 2-D projection. The metric decides the assignment: euclidean compares straight-line distance, cosine compares direction from the origin. No cluster boundary is drawn, because the boundary moves with the metric. No dimensionality reduction was run: this is a drawing, not a PCA or t-SNE projection. Feature scaling, other values of k and the discarded dimensions are not in the figure.

What to notice. The new point is 0.32 from centroid B and 0.47 from A, so euclidean distance assigns it to B. Cosine reads only direction from the origin: A lies on the same ray as the point and scores 1.00, while B scores 0.46, so a cosine k-means would assign it to A. The same three centroids give two answers, and no decision boundary is drawn, because it would move with the metric. The rank panel's top row is the whole assignment.

Text version

A scatter of three labelled cluster centroids and one new point in a hand-placed 2-D projection. No cluster or decision boundary is drawn. The default frame uses euclidean distance and k = 3: B is nearest at 0.32, A second at 0.47 and C third at 1.10, so the rank panel's first row, which is the assignment, is B. The metric control switches to cosine, where A scores 1.00 because it lies on the same ray from the origin as the new point, B scores 0.46 and C minus 0.98, so the assignment becomes A. The two metric buttons are the only control; the three centroids are always ranked together, so the top row is the assignment.

  1. Scale features first, since distance-based methods treat an unscaled column as more important.
  2. Choose the method from the assumed shape: k-means for compact blobs, density methods for arbitrary shapes, hierarchical for nested structure.
  3. For dimensionality reduction, decide whether you need global variance (PCA) or local neighbourhoods (t-SNE, UMAP).
  4. Select k or the neighbourhood size by stability across resamples, not by a single internal score.
  5. Validate by labelling a sample of cluster members and checking whether the labels agree within clusters.
  6. Project the result back to the original features and check that the grouping corresponds to something interpretable.

Trade-offs

ChoiceGood forCostFails when
k-meansFast, simple, scales to large dataAssumes spherical equal clusters, needs kClusters are elongated or unequal
Hierarchical clusteringNo fixed k, gives a dendrogramQuadratic memory, sensitive to linkage choiceThe dataset is large
PCALinear, invertible, preserves global varianceMisses nonlinear structureThe manifold is strongly curved
t-SNE or UMAPReveals local structure visuallyDistorts global distances, stochasticYou need quantitative distances
Trade-off mapInteractive

Good for

Fast, simple, scales to large data

Cost

Assumes spherical equal clusters, needs k

Fails when

Clusters are elongated or unequal

  1. Clusters are elongated or unequal
  2. The dataset is large
  3. The manifold is strongly curved
  4. You need quantitative distances

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: k-means is good for Fast, simple, scales to large data, costs Assumes spherical equal clusters, needs k, and fails when Clusters are elongated or unequal. Hierarchical clustering is good for No fixed k, gives a dendrogram, costs Quadratic memory, sensitive to linkage choice, and fails when The dataset is large. PCA is good for Linear, invertible, preserves global variance, costs Misses nonlinear structure, and fails when The manifold is strongly curved. t-SNE or UMAP is good for Reveals local structure visually, costs Distorts global distances, stochastic, and fails when You need quantitative distances

Failure modes

  • Clusters on unscaled features. One large-range column dominates the distance and the grouping reflects it alone. Debug: compare cluster assignments before and after standardisation.
  • Reading distances off a t-SNE plot. The gap between two clusters is interpreted as a magnitude. Debug: compute distances in the original space for points that look far apart in the embedding.
  • k chosen by eye. The number of clusters matches a desired story rather than the data. Debug: re-run with neighbouring k values and check whether membership is stable.

In production

Re-fit the projection on a schedule and monitor cluster population sizes, since drift often shows first as one segment absorbing another. Never serve decisions directly from an unvalidated cluster id; attach a labelled interpretation and a confidence check.

Common misconceptions

People think: A cluster is a real category in the data.

Actually: It is a partition produced by an algorithm and a parameter choice; whether it corresponds to anything meaningful requires external validation.

People think: UMAP and t-SNE preserve the structure of the data.

Actually: They preserve local neighbourhoods and distort global distances and cluster sizes, so visual separation is not evidence of distinct groups.

Interview angle

Distinguish grouping from projection, state the assumptions of each method, and insist on validating with labels or a downstream task. Warn about reading distances off a nonlinear embedding. Weak answers name algorithms without saying what shape they assume or how they would know the result is meaningful.

Sources