Why it matters
A classifier outputs a score, but decisions depend on how common each class actually is. Bayes' theorem makes that dependence explicit: the same likelihood ratio implies different posteriors when the prior changes. This is why a fraud model tuned on a balanced sample fails in production, and why a medical-style screening test with high sensitivity can still be wrong most of the time when the condition is rare.
Mental model
Bayes' theorem is updating a betting position as evidence arrives: you start with odds, the evidence has a known strength, and your new odds are the product. The metaphor breaks because the prior is not a subjective hunch in most ML uses — it is an estimated base rate, and the whole calculation is only as good as that estimate.
How it works
- 1Write down the prior probability of each class from the deployment population, not from the training sample.
- 2Estimate the likelihood of the observed features under each class, or the class-conditional density.
- 3Multiply prior by likelihood for each class to get unnormalised scores.
- 4Divide by the sum across classes so the posterior probabilities sum to one.
- 5Check calibration by comparing predicted probability with observed frequency in each bin.
- 6Recompute the prior when the deployment mix changes, since the likelihood terms usually stay fixed.
Step 1 of 6
Write down the prior probability of each class from the deployment population, not from the training sample.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write down the prior probability of each class from the deployment population, not from t… 2. Estimate the likelihood of the observed features under each class, or the class-condition… 3. Multiply prior by likelihood for each class to get unnormalised scores. 4. Divide by the sum across classes so the posterior probabilities sum to one. 5. Check calibration by comparing predicted probability with observed frequency in each bin. 6. Recompute the prior when the deployment mix changes, since the likelihood terms usually s…
- Write down the prior probability of each class from the deployment population, not from the training sample.
- Estimate the likelihood of the observed features under each class, or the class-conditional density.
- Multiply prior by likelihood for each class to get unnormalised scores.
- Divide by the sum across classes so the posterior probabilities sum to one.
- Check calibration by comparing predicted probability with observed frequency in each bin.
- Recompute the prior when the deployment mix changes, since the likelihood terms usually stay fixed.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Empirical prior | Matches the deployment base rate | Shifts whenever traffic mix changes | The population is unstable or unknown |
| Uniform prior | Simple, no estimation | Over-predicts rare classes | Rates are highly imbalanced |
| Naive independence assumption | Cheap, works with little data | Overconfident when features are correlated | Features are strongly dependent |
| Full probabilistic model | Correct uncertainty propagation | Expensive to specify and fit | Latency or data is limited |
Good for
Matches the deployment base rate
Cost
Shifts whenever traffic mix changes
Fails when
The population is unstable or unknown
- The population is unstable or unknown
- Rates are highly imbalanced
- Features are strongly dependent
- Latency or data is limited
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Empirical prior is good for Matches the deployment base rate, costs Shifts whenever traffic mix changes, and fails when The population is unstable or unknown. Uniform prior is good for Simple, no estimation, costs Over-predicts rare classes, and fails when Rates are highly imbalanced. Naive independence assumption is good for Cheap, works with little data, costs Overconfident when features are correlated, and fails when Features are strongly dependent. Full probabilistic model is good for Correct uncertainty propagation, costs Expensive to specify and fit, and fails when Latency or data is limited
Failure modes
- Base rate ignored at serving. A model trained on a balanced sample is deployed on a rare class and floods false positives. Debug: compare predicted positive rate against the known deployment prevalence.
- Prior double counting. Resampling to balance classes and then applying a prior again shifts probabilities twice. Debug: recompute posteriors from raw scores and compare with the pipeline's output.
- Correlated features in naive Bayes. Duplicate or derived features multiply the same evidence and produce extreme posteriors. Debug: check posterior confidence against accuracy in each confidence bucket.
In production
Monitor the predicted positive rate against the observed rate in labelled feedback, and recalibrate when the deployment mix changes. Store the prior used at training time with the model so a later recalibration is reproducible.
Common misconceptions
People think: A 99 percent accurate test means a positive result is 99 percent likely to be correct.
Actually: That depends on prevalence: with a rare condition, most positives can be false even at high accuracy, because the prior multiplies the likelihood.
People think: Bayesian reasoning requires subjective beliefs.
Actually: In applied ML the prior is usually an estimated frequency, and the theorem is just the rule for combining it with evidence.
Interview angle
A strong answer writes the rule, identifies each term with a concrete quantity, and works a rare-event example to show why the base rate dominates. It should connect to calibration and to naive Bayes' independence assumption. Weak answers state the formula and then interpret a positive test as the test's accuracy.
Sources
- Bishop, Pattern Recognition and Machine Learning — Bayesian inference and the prior-likelihood-posterior decomposition.
- Downey, Think Bayes — worked base-rate examples and updating.
- scikit-learn, Naive Bayes — the independence assumption and its practical behaviour.