Why it matters
A model's measured quality is only meaningful if the evaluation examples are genuinely unseen. Random row splits assume examples are exchangeable, which fails whenever rows share a user, a document or a time period. The engineering problem is that the split defines the question being answered: a random split estimates performance on more of the same, while a temporal or group split estimates performance on the future or on new entities.
Mental model
A split is a rehearsal versus the real performance: training is rehearsal, validation is a dress rehearsal, and test is opening night you only get once. The metaphor breaks because you can rerun opening night as many times as you like, and each rerun quietly turns it into another rehearsal.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Eleven synthetic training rows and one held-out row in a hand-placed 2-D feature layout. Rows are grouped by user: u7·a and u7·b belong to the same user as the held-out row. The metric is euclidean distance and k = 3; the dashed rings mark 0.25 and 0.50 from the held-out row. The axes are dimensionless, no unit applies, and the positions are drawn, not projected from a real table. A temporal split and a validation fold are not in the figure.
What to notice. The default frame is the row split: the two nearest rows are u7·a at 0.20 and u7·b at 0.35 — the held-out row's own user — and only the third, u10, is a different user. Step forward and the group rule removes every u7 row from training; the nearest remaining row is then u10 at 0.65. The score does not improve because the model changed; the question changed from more of the same user to a user it has never seen. A temporal split would change it again, and that ordering is not in this figure.
Text version
A scatter of eleven synthetic training rows and one held-out row labelled u7 held out, in a hand-placed 2-D layout with dashed distance rings at 0.25 and 0.50. The default frame applies a row split: the nearest rows are u7·a at 0.20 and u7·b at 0.35, two rows from the held-out row's own user, with u10 third at 0.65, all ringed and joined to the held-out row. Stepping forward to the user split dims the two u7 rows as held out and returns u10 at 0.65, u6 at 0.67 and u2 at 0.71. Three step controls walk from rows to row split to user split, and the rank panel follows the active rule.
- Identify the prediction unit at inference time. a future day, a new customer, an unseen document — and split on that boundary.
- Hold out a test set first and restrict access to it until the final decision.
- Split the remainder into train and validation, using the same grouping or temporal rule.
- Stratify by the label and by important segments so rare classes appear in every split.
- Fit all preprocessing on the training split only, then apply the fitted transform to the others.
- For small data, use grouped or time-series cross-validation and report the spread across folds, not one number.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Random split | Maximum data, simple, low variance | Optimistic when rows are correlated | Users, documents or time create dependence |
| Temporal split | Matches forecasting deployment | Less training data, one ordering only | The future distribution differs sharply |
| Group split | Honest estimate for new entities | Fewer groups than rows, higher variance | Groups are small or numerous |
| Cross-validation | Uses data efficiently, gives a spread | Expensive and easy to leak across folds | The pipeline is not refit inside each fold |
Good for
Maximum data, simple, low variance
Cost
Optimistic when rows are correlated
Fails when
Users, documents or time create dependence
- Users, documents or time create dependence
- The future distribution differs sharply
- Groups are small or numerous
- The pipeline is not refit inside each fold
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Random split is good for Maximum data, simple, low variance, costs Optimistic when rows are correlated, and fails when Users, documents or time create dependence. Temporal split is good for Matches forecasting deployment, costs Less training data, one ordering only, and fails when The future distribution differs sharply. Group split is good for Honest estimate for new entities, costs Fewer groups than rows, higher variance, and fails when Groups are small or numerous. Cross-validation is good for Uses data efficiently, gives a spread, costs Expensive and easy to leak across folds, and fails when The pipeline is not refit inside each fold
Failure modes
- Duplicate rows across splits. Near-identical examples appear in train and test and the score is inflated. Debug: hash normalised text or features and count exact and near-duplicate collisions across splits.
- Preprocessing before the split. Scaler or target encoding statistics include validation data. Debug: refit the transform inside a pipeline on the training fold only and compare scores.
- Test set reused for tuning. Reported performance converges to validation performance and stops predicting deployment. Debug: count how many times the test metric has been read and compare with the untouched baseline.
In production
Version the split definition with the model, since a changed split makes two runs incomparable. Hold back a rolling out-of-time slice and re-score on it after deployment, because a static test set ages as the world moves.
Common misconceptions
People think: A random 80/20 split is always valid.
Actually: It assumes rows are independent; when they share users, documents or time, the test set is effectively seen during training.
People think: Cross-validation removes the need for a holdout.
Actually: Cross-validation estimates variance across folds of the same data; it does not protect against tuning on that data, so a final untouched set is still needed.
Interview angle
A strong answer derives the split from the deployment unit, then discusses stratification, preprocessing isolation and test-set discipline. It should name the leak each split prevents. Weak answers say '80/20 random' without asking whether rows are independent or what the model will see at inference.
Sources
- scikit-learn, Cross-validation: evaluating estimator performance — grouped, stratified and time-series split strategies.
- scikit-learn, Common pitfalls and recommended practices — leakage from preprocessing outside the split.
- scikit-learn, TimeSeriesSplit — a splitter that respects temporal order.