Why it matters
The same features can support either task, and the choice is about what decision the product needs. Predicting a probability and then thresholding is different from predicting a number and comparing it to a cutoff: the first can be calibrated and tuned per cost, the second preserves magnitude. Confusing the two produces models that are technically trained but operationally unusable, such as a churn score with no threshold or a class label where the size of the value mattered.
Mental model
Classification is sorting mail into bins; regression is writing the weight on the envelope. The metaphor breaks because the bins can be ordered and the weight can be bucketed, so the boundary is a product decision about what you need to report, not a property of the data.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Eight synthetic labelled accounts in a two-feature standardised space: usage and tenure, each in standard deviations from the training mean, so the axes carry σ as their unit. The nearest labelled rows decide a new account's class, and k is how many rows vote. No decision boundary is drawn, because a k-nearest-neighbour label has no boundary until the vote is taken. A regressor reading the same rows would return a number; that head is not in the figure.
What to notice. The new account is 0.14 from the nearest row, which is churn, and 0.18 and 0.26 from two stay rows. At k = 1 the label is churn; at k = 3 the majority is stay; at k = 5 three of five are stay. The rows never move, so the flip belongs to the rule, not to the data. At k = 2 the vote is one each and the rule has no majority, so the tie-break is a design choice rather than a property of the geometry. Standardising the two columns is what lets usage and tenure share one axis; the scaler itself is not shown.
Text version
A scatter of eight labelled synthetic accounts on two standardised axes, usage and tenure in standard deviations. The default frame uses euclidean distance and k = 3: the new account in the centre is joined to its three nearest rows, churn a1 at 0.14, stay a2 at 0.18 and stay a3 at 0.26, all ringed, and the rank panel lists them in that order, so two of three vote stay. A slider sets k from one to five; at k = 1 only churn a1 is returned and the label is churn, and at k = 2 the two returned rows split one each. No boundary line is drawn anywhere in the plot.
- Decide what the downstream action consumes: a ranked score, a hard label, or an estimated quantity.
- Match the head to the target: sigmoid for binary, softmax for mutually exclusive classes, linear for continuous.
- Choose the loss from the likelihood. binary or categorical cross-entropy, or squared error for a Gaussian target.
- For classification, tune the decision threshold on validation data using the real cost of each error type.
- For regression, report an error metric in the target's units and inspect residuals for structure.
- For ordinal targets, use an ordinal model or cumulative link rather than collapsing to either extreme.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Binary classification | Calibrated probability, adjustable threshold | Discards magnitude information | The size of the effect drives the decision |
| Regression | Preserves magnitude and ranking | Needs a separate cutoff for actions | The cost of over- and under-prediction is asymmetric |
| Ordinal classification | Respects order without assuming spacing | More complex models and evaluation | You have very few examples per level |
| Thresholded regression | Reuses an existing model | Calibration is lost and thresholds are arbitrary | Probabilities are needed downstream |
Good for
Calibrated probability, adjustable threshold
Cost
Discards magnitude information
Fails when
The size of the effect drives the decision
- The size of the effect drives the decision
- The cost of over- and under-prediction is asymmetric
- You have very few examples per level
- Probabilities are needed downstream
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Binary classification is good for Calibrated probability, adjustable threshold, costs Discards magnitude information, and fails when The size of the effect drives the decision. Regression is good for Preserves magnitude and ranking, costs Needs a separate cutoff for actions, and fails when The cost of over- and under-prediction is asymmetric. Ordinal classification is good for Respects order without assuming spacing, costs More complex models and evaluation, and fails when You have very few examples per level. Thresholded regression is good for Reuses an existing model, costs Calibration is lost and thresholds are arbitrary, and fails when Probabilities are needed downstream
Failure modes
- Class labels used where magnitude matters. The model cannot distinguish a near miss from a catastrophic value. Debug: compare the business cost of errors within the same predicted class.
- Threshold left at the default. A model trained for balanced accuracy is deployed with a 0.5 cutoff on imbalanced traffic. Debug: sweep the threshold against the confusion matrix at the deployment prevalence.
- Regression on a bounded target. Predictions fall outside the valid range and are clipped after the fact. Debug: check the support of predictions against the label range.
In production
Store the decision threshold with the model and monitor the predicted positive rate, since prevalence drift changes the meaning of a fixed cutoff. For regression, monitor residual bias by segment, because a model can have low average error and systematic error in one group.
Common misconceptions
People think: Regression is just classification with more classes.
Actually: The output space and loss differ: regression assumes an order and a metric on values, while classification treats labels as exchangeable categories.
People think: A classifier's score is a probability.
Actually: It is only a probability if calibrated; many models produce scores that rank well but are systematically overconfident.
Interview angle
A strong answer starts from the decision the product needs, then derives head, loss, threshold and metric from it. It should discuss ordinal targets and calibration. Weak answers say classification is for categories and regression for numbers without saying how the threshold or the error cost changes the design.
Sources
- scikit-learn, Supervised learning — classifier and regressor families and their outputs.
- Hastie, Tibshirani & Friedman, The Elements of Statistical Learning — the decision-theoretic view of classification and regression.
- Bishop, Pattern Recognition and Machine Learning — discriminative versus predictive distributions.