Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 19 of 27

Feature engineering

0 of 27 complete

Why it matters

Models see numbers, not records. The gap between a raw table and a useful input is where domain knowledge enters: a timestamp becomes time since last activity, a category becomes a target statistic, two columns become a ratio. With limited data this matters more than architecture. The cost is that each derived feature creates a training-serving contract that must be maintained or the model silently receives different inputs in production.

Mental model

Feature engineering is writing the question in a language the model can answer: the same information, restated so the pattern is easier to see. The metaphor breaks because the restatement can also destroy information — binning a continuous variable or dropping a rare category is irreversible, and the model cannot recover what the transform removed.

How it works

A ratio is a direction, not a columnInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Six synthetic orders plotted by order value and discount. Both columns are divided by the same constant — ₹1 lakh — so the plotted coordinates are unitless and the angle from the origin is the discount rate. Cosine scores that angle; euclidean scores raw size. The engineered feature is the ratio, and the metric decides whether the index reads it. Dividing each column by its own scale would change the angle, which is why a ratio needs one unit; encoding, missing-value fills and train-serve parity are not in the figure.

What to notice. Under cosine the two orders at rate 0.20 tie at 1.00 although one is about half the query's size and the other about one and a half times it. Under euclidean those two fall to third and fourth: the nearest rows are 0.09 and 0.19 away and match the size, at rates of 0.05 and 0.50. Same six points, two questions — the same discount rate, or the same order size — and the metric is what chooses. The rings appear only under euclidean, where distance is the question.

Text version

A scatter of six synthetic orders on two unitless axes, order value and discount, both divided by the same constant. The default frame uses cosine and k = 3: the new order at 0.60 of value and 0.12 of discount is joined to its three nearest by angle, the two same-rate orders at 1.00 and a small order at 0.99, and the rank panel lists them in that order. The two same-rate orders lie on the query's ray and differ in size. A metric control switches to euclidean, where dashed rings at 0.25 and 0.50 appear and the same-size orders at 0.09 and 0.19 replace them. A slider sets k from one to six.

  1. Profile each column: type, missingness, cardinality, range and how it is produced upstream.
  2. Encode categoricals with a method matched to cardinality. one-hot for few levels, target or hashing for many.
  3. Scale numeric features for distance-based and gradient-based models, fitting the scaler on training data only.
  4. Derive time features from the prediction moment, such as recency and windowed aggregates, using only prior data.
  5. Add interactions only where domain knowledge suggests a joint effect, then measure whether they help.
  6. Package the transforms in one fitted object so training and serving apply identical logic.

Trade-offs

ChoiceGood forCostFails when
One-hot encodingNo order assumed, interpretableDimension grows with cardinalityCategories are high-cardinality
Target encodingCompact, captures label relationshipLeaks unless computed out-of-foldRare categories or small data
StandardisationHelps gradient-based models convergeSensitive to outliers, must be persistedThe feature is heavy-tailed
Hand-built interactionsEncodes domain knowledge directlyCombinatorial growth and overfitting riskThe feature set is already large
Trade-off mapInteractive

Good for

No order assumed, interpretable

Cost

Dimension grows with cardinality

Fails when

Categories are high-cardinality

  1. Categories are high-cardinality
  2. Rare categories or small data
  3. The feature is heavy-tailed
  4. The feature set is already large

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: One-hot encoding is good for No order assumed, interpretable, costs Dimension grows with cardinality, and fails when Categories are high-cardinality. Target encoding is good for Compact, captures label relationship, costs Leaks unless computed out-of-fold, and fails when Rare categories or small data. Standardisation is good for Helps gradient-based models converge, costs Sensitive to outliers, must be persisted, and fails when The feature is heavy-tailed. Hand-built interactions is good for Encodes domain knowledge directly, costs Combinatorial growth and overfitting risk, and fails when The feature set is already large

Failure modes

  • Encoder fitted on all data. Validation statistics enter the transform and scores inflate. Debug: refit inside a pipeline per fold and compare the score with the global fit.
  • Train-serve skew. A feature is computed with a different window or join online and offline. Debug: log feature values at serving time and compare their distribution with the training set for the same entity.
  • Silent missing-value fill. A default value like zero is indistinguishable from a real measurement. Debug: add an explicit missing indicator and check its coefficient or importance.

In production

Store the fitted transformer with the model and version it, since a re-fitted scaler changes every input. Monitor per-feature distributions online against training baselines, because skew usually appears in one feature before it appears in the metric.

Common misconceptions

People think: Deep learning removes the need for feature engineering.

Actually: It reduces manual feature construction for raw perceptual data, but tabular and business data still depend on encoding, aggregation and time alignment.

People think: More features always improve a model.

Actually: Uninformative or leaking features add variance, cost and maintenance, and can degrade performance while making the model harder to debug.

Interview angle

A strong answer ties each transform to model family and data cardinality, insists on fitting inside the split, and treats serving parity as part of the feature's definition. It should give a concrete skew example. Weak answers list encodings without saying which models need them or how the transform is kept consistent in production.

Sources