Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 18 of 27

Data leakage

0 of 27 complete

Why it matters

A model is only as honest as its evaluation, and leakage inflates that evaluation while leaving the code looking correct. It is insidious because leaked models often show excellent validation scores and fail only in deployment, after the cost of training and integration. Detecting it requires asking, for every feature, whether the value would exist at the moment a prediction must be made.

Mental model

Leakage is reading tomorrow's newspaper to forecast tomorrow's weather: the forecast is perfect and useless. The metaphor breaks because the leak is rarely a single obvious column — it is often an aggregate, an id, or a timestamp that encodes the outcome indirectly.

How it works

A clean boundary, an unclean distanceInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Two synthetic split regions in a hand-placed 2-D feature layout: seven training rows inside the train ellipse, and three test rows plus the row being predicted inside the test ellipse. The dashed ellipses are drawn extents, not a model boundary. Euclidean distance, k = 3; the dashed ring marks 0.50. The axes are dimensionless and carry no unit. Preprocessing leaks and post-outcome features are not in the figure.

What to notice. The row being predicted is 0.07 from a row in the training split, and that row is a near-copy: the same fields and values, with the free-text field spelled differently. The split is correct — the two copies are on opposite sides of the boundary by construction — and the estimate is still optimistic, because the distance the model has to generalise over is 0.07, not the gap between the ellipses. Deduplicating before the split removes the copy; moving the boundary does not.

Text version

A scatter of seven synthetic training rows and three test rows in two dashed regions labelled train and test, with the row being predicted near the boundary. The default frame uses euclidean distance and k = 3: the nearest row is labelled train copy and sits 0.07 away, marked with a callout, followed by a test row at 0.18 and a training row at 0.34. The copy sits inside the train ellipse and the query inside the test ellipse, so the split is clean while the content is shared. The rank panel lists the three nearest rows and their distances; the ellipses are hand-placed extents. Three step controls walk from the unmarked rows, to the drawn split with its ranked neighbours, to the annotation on the copy.

  1. Write down the exact prediction moment and list which fields would be populated then.
  2. Remove or lag every feature that is written after that moment, including aggregates computed over the full history.
  3. Split before any fitting, and put all preprocessing inside a pipeline that is refit per fold.
  4. Check identifiers and timestamps for proxy leakage: sequential ids often encode acquisition order and therefore label prevalence.
  5. Deduplicate before splitting, using hashes on normalised content rather than exact strings.
  6. Compare a model trained with and without the suspect feature; a large drop is evidence the feature carried the label.

Trade-offs

ChoiceGood forCostFails when
Aggressive feature removalSafer, closer to deploymentMay discard genuinely predictive signalThe feature really is available at prediction time
Pipeline-enforced splitsPrevents preprocessing leaks structurallyMore code and slower experimentationThe team works outside the pipeline
Temporal validationMatches forecasting realityLess data and higher varianceThe product is not time-dependent
Manual review of featuresCatches semantic leaks automation missesDoes not scale and depends on domain knowledgeFeature count is large or ownership is unclear
Trade-off mapInteractive

Good for

Safer, closer to deployment

Cost

May discard genuinely predictive signal

Fails when

The feature really is available at prediction time

  1. The feature really is available at prediction time
  2. The team works outside the pipeline
  3. The product is not time-dependent
  4. Feature count is large or ownership is unclear

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Aggressive feature removal is good for Safer, closer to deployment, costs May discard genuinely predictive signal, and fails when The feature really is available at prediction time. Pipeline-enforced splits is good for Prevents preprocessing leaks structurally, costs More code and slower experimentation, and fails when The team works outside the pipeline. Temporal validation is good for Matches forecasting reality, costs Less data and higher variance, and fails when The product is not time-dependent. Manual review of features is good for Catches semantic leaks automation misses, costs Does not scale and depends on domain knowledge, and fails when Feature count is large or ownership is unclear

Failure modes

  • Post-outcome feature. A field recorded after the label event predicts almost perfectly. Debug: check the feature's write timestamp against the label timestamp for a sample of rows.
  • Duplicate rows across splits. The same example appears in train and test after cleaning differences. Debug: hash normalised content and count collisions between splits.
  • Full-history aggregate. A mean computed over all dates includes the prediction window. Debug: recompute the feature using only data before each prediction time and compare distributions.

In production

Recompute features from a strict as-of join in the serving path and compare them against training-time values for the same entities. Alert when a feature's importance is implausibly high or its distribution differs sharply between offline and online computation.

Common misconceptions

People think: Leakage is always a preprocessing bug.

Actually: It is often semantic: a feature that is legitimately in the table but only populated after the outcome is known.

People think: A very high validation score is a good sign.

Actually: Scores near perfection usually indicate leakage; real problems with noisy labels rarely reach them.

Interview angle

A strong answer defines leakage by the prediction moment, gives the main families, and proposes a concrete check for each. It should treat an implausibly high score as a warning. Weak answers mention 'using test data in training' and stop, missing target, group and temporal leakage.

Sources