Why it matters
Instruction tuning and preference tuning both want more coverage than annotators can supply. A capable model can draft examples across many task types, which makes dataset size a budget question rather than a scheduling one. The risk is that the generator's errors become the student's ground truth, and that repeated generation on model output narrows the distribution instead of widening it.
Mental model
Synthetic data is photocopying a photocopy: fast, convenient, and each generation loses a little definition until the text is no longer readable. The metaphor breaks because you can inspect each copy and mix it back with the originals, which is the practice that keeps a synthetic set usable.
How it works
- 1Write a seed set of human examples that define the behaviour and format you want replicated.
- 2Generate variations with an explicit schema and a diversity instruction, sampling across categories rather than one prompt template.
- 3Filter aggressively: verify factual claims against a source, run executable checks where possible, and discard refusals and malformed records.
- 4Deduplicate against both the seed set and the generated set, since generators repeat themselves.
- 5Mix synthetic and human data rather than replacing one with the other, and record the ratio.
- 6Evaluate on human-written held-out prompts only, so the score is not measuring the generator.
Step 1 of 6
Write a seed set of human examples that define the behaviour and format you want replicated.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write a seed set of human examples that define the behaviour and format you want replicat… 2. Generate variations with an explicit schema and a diversity instruction, sampling across … 3. Filter aggressively: verify factual claims against a source, run executable checks where … 4. Deduplicate against both the seed set and the generated set, since generators repeat them… 5. Mix synthetic and human data rather than replacing one with the other, and record the rat… 6. Evaluate on human-written held-out prompts only, so the score is not measuring the genera…
- Write a seed set of human examples that define the behaviour and format you want replicated.
- Generate variations with an explicit schema and a diversity instruction, sampling across categories rather than one prompt template.
- Filter aggressively: verify factual claims against a source, run executable checks where possible, and discard refusals and malformed records.
- Deduplicate against both the seed set and the generated set, since generators repeat themselves.
- Mix synthetic and human data rather than replacing one with the other, and record the ratio.
- Evaluate on human-written held-out prompts only, so the score is not measuring the generator.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Fully synthetic | Scale and coverage | Inherited errors and mode collapse | Correctness is the product |
| Synthetic plus human seed | Broad with an anchor | More pipeline work | You cannot verify outputs |
| Self-distillation rounds | Cheap difficulty escalation | Amplifies existing failures | The base model is already wrong |
| Human only | Trustworthy targets | Slow and expensive | Coverage is the bottleneck |
Good for
Scale and coverage
Cost
Inherited errors and mode collapse
Fails when
Correctness is the product
- Correctness is the product
- You cannot verify outputs
- The base model is already wrong
- Coverage is the bottleneck
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Fully synthetic is good for Scale and coverage, costs Inherited errors and mode collapse, and fails when Correctness is the product. Synthetic plus human seed is good for Broad with an anchor, costs More pipeline work, and fails when You cannot verify outputs. Self-distillation rounds is good for Cheap difficulty escalation, costs Amplifies existing failures, and fails when The base model is already wrong. Human only is good for Trustworthy targets, costs Slow and expensive, and fails when Coverage is the bottleneck
Failure modes
- Model collapse. Later generations are less diverse and the student parrots one style. Debug: measure distinct-n or embedding spread per generation round and compare with the seed set.
- Error amplification. A systematic mistake appears in most synthetic answers and becomes the student's belief. Debug: sample by category and check claims against a trusted source, not against another model.
- Contamination. Synthetic evaluation items were generated from the same prompts as training. Debug: hash prompts and keep evaluation prompts human-authored and disjoint.
In production
Tag every record with its generator and round, and keep the ability to retrain without a given batch. Monitor output diversity and refusal rate after each synthetic-heavy run.
Common misconceptions
People think: Synthetic data is free data.
Actually: Verification and filtering usually cost more than generation.
People think: A stronger generator fixes quality.
Actually: A stronger generator produces more convincing errors, which are harder to filter.
Interview angle
Treat generation as the cheap half and verification as the work. Name diversity collapse and error amplification, and say evaluation must use human prompts. Weak answers describe a pipeline that generates and trains with no filtering step and no provenance.
Sources
- Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions — bootstrapping instruction data from a seed set.
- Honovich et al., Unnatural Instructions — generated instructions as a substitute for human annotation.
- Shumailov et al., The Curse of Recursion: Training on Generated Data Makes Models Forget — degradation from training on recursively generated data.