Why it matters
A churn classifier returns a number in a known range, so you can measure calibration, choose a threshold and bound the damage of an error. A model writing a customer reply can produce anything, including a correct-sounding falsehood, and no threshold contains it. Treating the second like the first produces evaluation that measures nothing and guardrails that arrive too late.
Mental model
A predictive model is a multiple-choice exam; a generative model is an essay question. The metaphor breaks because an essay is marked by a reader who understands the subject, while automated grading of generated text is itself a model judgement with its own error.
How it works
- 1State the output contract: a label set, a numeric range, or a specification for the generated artefact.
- 2For predictive tasks, build a labelled set and measure calibration and per-class errors.
- 3For generative tasks, define a rubric and collect reference outputs where possible.
- 4Constrain generation with schemas, allowed values and post-generation validation.
- 5Evaluate both with slices, since aggregate scores hide the class or topic that fails.
- 6Decide where a deterministic check can replace a model judgement, and put it first.
Step 1 of 6
State the output contract: a label set, a numeric range, or a specification for the generated artefact.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. State the output contract: a label set, a numeric range, or a specification for the gener… 2. For predictive tasks, build a labelled set and measure calibration and per-class errors. 3. For generative tasks, define a rubric and collect reference outputs where possible. 4. Constrain generation with schemas, allowed values and post-generation validation. 5. Evaluate both with slices, since aggregate scores hide the class or topic that fails. 6. Decide where a deterministic check can replace a model judgement, and put it first.
- State the output contract: a label set, a numeric range, or a specification for the generated artefact.
- For predictive tasks, build a labelled set and measure calibration and per-class errors.
- For generative tasks, define a rubric and collect reference outputs where possible.
- Constrain generation with schemas, allowed values and post-generation validation.
- Evaluate both with slices, since aggregate scores hide the class or topic that fails.
- Decide where a deterministic check can replace a model judgement, and put it first.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Predictive model | Calibrated, cheap, easy to threshold | Only answers predefined questions | The output space is open |
| Generative model | Handles open-ended input and output | Evaluation is indirect and costly | The decision must be a fixed label |
| Generator with schema | Structured, validatable output | Constrains expression and can truncate | The task needs free-form prose |
| Classifier plus generator | Cheap routing, rich explanation | Two systems and a join between them | Consistency between the two matters |
Good for
Calibrated, cheap, easy to threshold
Cost
Only answers predefined questions
Fails when
The output space is open
- The output space is open
- The decision must be a fixed label
- The task needs free-form prose
- Consistency between the two matters
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Predictive model is good for Calibrated, cheap, easy to threshold, costs Only answers predefined questions, and fails when The output space is open. Generative model is good for Handles open-ended input and output, costs Evaluation is indirect and costly, and fails when The decision must be a fixed label. Generator with schema is good for Structured, validatable output, costs Constrains expression and can truncate, and fails when The task needs free-form prose. Classifier plus generator is good for Cheap routing, rich explanation, costs Two systems and a join between them, and fails when Consistency between the two matters
Failure modes
- Classification metric on generated text. Exact match scores near zero and says nothing about quality. Debug: check whether the metric distinguishes a good paraphrase from a wrong answer.
- Unbounded output. The generator emits content outside the intended form or policy. Debug: validate every output against the schema and sample the rejects.
- Label set drift. New categories appear in production and the classifier forces them into old ones. Debug: inspect the confidence distribution and the share of low-confidence predictions.
In production
For predictive models track calibration and per-class error; for generative ones track schema validity, refusal rate and rubric scores on a sample. Both need input drift monitoring.
Common misconceptions
People think: Generative models replace predictive ones.
Actually: A classifier is cheaper, calibrated and easier to audit when the question is closed; generation is for open output.
People think: Generative output cannot be evaluated.
Actually: It can, with rubrics, references and deterministic checks on form; the evaluation is just less direct.
Interview angle
Strong answers start from the output space and derive the evaluation and constraints from it, then say when a classifier is the better tool. Weak answers treat generation as a strictly newer version of prediction.
Sources
- Brown et al., Language Models are Few-Shot Learners — open-ended generation from a prompt rather than a fixed label set.
- Karpathy, Software 2.0 — learned functions with different failure characteristics from explicit code.
- scikit-learn user guide — classification metrics, calibration and threshold selection.