Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 8 of 27

Distributions

0 of 27 complete

Why it matters

A model's output layer and loss both assume a distribution. Using squared error assumes Gaussian noise; using cross-entropy assumes a categorical distribution; using a Poisson likelihood assumes counts with variance equal to the mean. When the assumption is wrong, the model is not merely suboptimal — it is confident in the wrong places. Choosing a family is therefore a statement about the data-generating process, and it can be checked.

Mental model

A distribution family is a template for the shape of randomness: pick the template, then fit its knobs to the data. The metaphor breaks because real data often needs a mixture or a transformation before any template fits, and a poor template can still produce a plausible-looking histogram.

How it works

Mechanism, step by stepInteractive
  1. 1Identify the support first: binary, multi-class, non-negative count, positive continuous, or unbounded real.
  2. 2Plot the empirical distribution and compare its skew and tails against candidate families.
  3. 3Estimate parameters by maximum likelihood, which for a Gaussian reduces to mean and variance.
  4. 4Check the fitted distribution against the data with quantile plots or a calibration check on held-out data.
  5. 5Transform when the support does not match, such as modelling a log of positive skewed values.
  6. 6Use a mixture or a more flexible family when a single template leaves systematic residuals.

Step 1 of 6

Identify the support first: binary, multi-class, non-negative count, positive continuous, or unbounded real.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Identify the support first: binary, multi-class, non-negative count, positive continuous,… 2. Plot the empirical distribution and compare its skew and tails against candidate families. 3. Estimate parameters by maximum likelihood, which for a Gaussian reduces to mean and varia… 4. Check the fitted distribution against the data with quantile plots or a calibration check… 5. Transform when the support does not match, such as modelling a log of positive skewed val… 6. Use a mixture or a more flexible family when a single template leaves systematic residual…

  1. Identify the support first: binary, multi-class, non-negative count, positive continuous, or unbounded real.
  2. Plot the empirical distribution and compare its skew and tails against candidate families.
  3. Estimate parameters by maximum likelihood, which for a Gaussian reduces to mean and variance.
  4. Check the fitted distribution against the data with quantile plots or a calibration check on held-out data.
  5. Transform when the support does not match, such as modelling a log of positive skewed values.
  6. Use a mixture or a more flexible family when a single template leaves systematic residuals.

Trade-offs

ChoiceGood forCostFails when
GaussianClosed forms, easy gradients, familiar intervalsUnbounded support, thin tailsData is positive, bounded or heavy-tailed
PoissonCorrect for counts, variance tied to meanFails when counts are overdispersedVariance exceeds the mean
Log-normalPositive skewed values, multiplicative noiseParameters are less interpretableThe data has a hard upper bound
Empirical distributionNo shape assumptionDoes not generalise beyond observed valuesYou need to sample unseen outcomes
Trade-off mapInteractive

Good for

Closed forms, easy gradients, familiar intervals

Cost

Unbounded support, thin tails

Fails when

Data is positive, bounded or heavy-tailed

  1. Data is positive, bounded or heavy-tailed
  2. Variance exceeds the mean
  3. The data has a hard upper bound
  4. You need to sample unseen outcomes

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Gaussian is good for Closed forms, easy gradients, familiar intervals, costs Unbounded support, thin tails, and fails when Data is positive, bounded or heavy-tailed. Poisson is good for Correct for counts, variance tied to mean, costs Fails when counts are overdispersed, and fails when Variance exceeds the mean. Log-normal is good for Positive skewed values, multiplicative noise, costs Parameters are less interpretable, and fails when The data has a hard upper bound. Empirical distribution is good for No shape assumption, costs Does not generalise beyond observed values, and fails when You need to sample unseen outcomes

Failure modes

  • Gaussian assumed for counts. The model predicts negative counts and confidence intervals that cross zero. Debug: check the support of predictions against the support of the labels.
  • Overdispersion ignored. Poisson intervals are too narrow because variance exceeds the mean. Debug: compare sample variance with sample mean within groups.
  • Tail behaviour unexamined. A mean-based metric is dominated by a few extreme values. Debug: compare mean and median and inspect the top percentile.

In production

Monitor the distribution of predictions and residuals, not just their averages, since a change in shape precedes a change in error. Cap or transform outputs whose support does not match the label space, and alert when predicted values fall outside it.

Common misconceptions

People think: Most real-world data is normally distributed.

Actually: Many quantities are skewed, bounded or heavy-tailed; the Gaussian is common as an assumption because it is tractable, not because it is universal.

People think: The distribution choice only affects interpretation.

Actually: It defines the likelihood, so it changes the loss, the gradients and the uncertainty the model reports.

Interview angle

A strong answer starts from the support and the data-generating process, maps families to output layers and losses, and names a diagnostic for misfit. It should discuss tails and overdispersion. Weak answers list distributions without connecting them to a likelihood or saying how a wrong choice shows up.

Sources