Why it matters
Every dashboard, evaluation report and data audit reduces a distribution to a handful of numbers. The reduction is where most misinterpretation enters: a mean latency hides the tail users feel, a variance on a bimodal sample describes a gap that no observation sits in, and an unweighted average over segments misrepresents the whole. Knowing which summary is robust to what is a prerequisite for reading any metric.
Mental model
Descriptive statistics are the headline of a dataset: a few words that tell you what it is broadly like. The metaphor breaks because headlines can be technically true and deeply misleading, and unlike a headline you can compute several of them to check whether they agree.
How it works
- 1Count missing values and inspect the distribution before computing any centre, since a mean over a filtered subset is a different statistic.
- 2Report the median alongside the mean; a large gap signals skew or outliers.
- 3Report a spread measure with the centre, because a mean without dispersion says nothing about individual cases.
- 4Use quantiles for skewed operational metrics, especially p50, p95 and p99 for latency.
- 5Break the summary down by segment and compare with the pooled value to detect aggregation effects.
- 6State the denominator and the time window for every number, since rates without them are ambiguous.
Step 1 of 6
Count missing values and inspect the distribution before computing any centre, since a mean over a filtered subset is a different statistic.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Count missing values and inspect the distribution before computing any centre, since a me… 2. Report the median alongside the mean; a large gap signals skew or outliers. 3. Report a spread measure with the centre, because a mean without dispersion says nothing a… 4. Use quantiles for skewed operational metrics, especially p50, p95 and p99 for latency. 5. Break the summary down by segment and compare with the pooled value to detect aggregation… 6. State the denominator and the time window for every number, since rates without them are …
- Count missing values and inspect the distribution before computing any centre, since a mean over a filtered subset is a different statistic.
- Report the median alongside the mean; a large gap signals skew or outliers.
- Report a spread measure with the centre, because a mean without dispersion says nothing about individual cases.
- Use quantiles for skewed operational metrics, especially p50, p95 and p99 for latency.
- Break the summary down by segment and compare with the pooled value to detect aggregation effects.
- State the denominator and the time window for every number, since rates without them are ambiguous.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Mean and standard deviation | Uses all data, simple algebra | Distorted by outliers and skew | The distribution has a heavy tail |
| Median and IQR | Robust to outliers, describes the typical case | Ignores tail severity, harder to combine across groups | Worst-case behaviour is the concern |
| Quantiles | Captures tails directly | Noisy at extreme percentiles with small samples | Sample size per bucket is small |
| Histogram | Shows shape and multimodality | Harder to compare across runs | You need a single decision metric |
Good for
Uses all data, simple algebra
Cost
Distorted by outliers and skew
Fails when
The distribution has a heavy tail
- The distribution has a heavy tail
- Worst-case behaviour is the concern
- Sample size per bucket is small
- You need a single decision metric
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Mean and standard deviation is good for Uses all data, simple algebra, costs Distorted by outliers and skew, and fails when The distribution has a heavy tail. Median and IQR is good for Robust to outliers, describes the typical case, costs Ignores tail severity, harder to combine across groups, and fails when Worst-case behaviour is the concern. Quantiles is good for Captures tails directly, costs Noisy at extreme percentiles with small samples, and fails when Sample size per bucket is small. Histogram is good for Shows shape and multimodality, costs Harder to compare across runs, and fails when You need a single decision metric
Failure modes
- Mean hides a bimodal split. The average sits between two clusters and describes no actual user. Debug: plot the histogram and check whether the mean falls in a low-density region.
- Unweighted segment average. Pooling per-segment means equally gives a number that matches no population. Debug: recompute the pooled mean with sample-size weights and compare.
- Percentile of percentiles. Averaging per-window p95 values does not produce the overall p95. Debug: recompute from raw observations and compare against the aggregated figure.
In production
Publish p50, p95 and p99 with the sample count for any latency or quality metric, and never aggregate percentiles by averaging them. Alert on distribution shift in the median and in the tail separately, because they move for different reasons.
Common misconceptions
People think: The mean is the most objective summary.
Actually: It is one choice among several and is dominated by extreme values; the median is often the better description of a typical case.
People think: A standard deviation tells you the range.
Actually: It describes spread under an approximately symmetric distribution; for skewed data it says little about the tails or the maximum.
Interview angle
A strong answer chooses a summary from the question and the shape of the data, and names the failure of each: mean under skew, variance under bimodality, averaged percentiles. It should mention denominators and segment weighting. Weak answers recite mean, median and mode without saying when each misleads.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods — measures of location, scale and robustness.
- OpenIntro Statistics — summary statistics and distribution shape.
- pandas, DataFrame.describe — the standard summary output and its quantiles.