Why it matters
When a model's output becomes an input to future decisions, the data stops being an independent sample. A recommender that only shows popular items observes clicks only on popular items; an assistant that drafts tickets creates the corpus it will later retrieve from; an agent that writes files leaves artefacts that change its next run. The loop can improve the product and quietly destroy the evidence needed to tell whether it did.
Mental model
It is an echo in a room: the sound is real, but each reflection reinforces some frequencies and cancels others. The analogy breaks because an echo fades, while a data loop accumulates and can persist after the original cause is gone.
How it works
- 1Map where model output enters future inputs: training data, retrieval corpus, ranking features, prompt examples.
- 2Keep a holdout that is never influenced by model output, and use it for evaluation.
- 3Randomise a small share of exposures so unexplored options still generate data.
- 4Log exposure and outcome separately, so a click can be attributed to what was shown.
- 5Track diversity and coverage alongside accuracy, since narrowing is the first symptom.
- 6Audit generated content before it enters a corpus the model will retrieve from.
Step 1 of 6
Map where model output enters future inputs: training data, retrieval corpus, ranking features, prompt examples.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Map where model output enters future inputs: training data, retrieval corpus, ranking fea… 2. Keep a holdout that is never influenced by model output, and use it for evaluation. 3. Randomise a small share of exposures so unexplored options still generate data. 4. Log exposure and outcome separately, so a click can be attributed to what was shown. 5. Track diversity and coverage alongside accuracy, since narrowing is the first symptom. 6. Audit generated content before it enters a corpus the model will retrieve from.
- Map where model output enters future inputs: training data, retrieval corpus, ranking features, prompt examples.
- Keep a holdout that is never influenced by model output, and use it for evaluation.
- Randomise a small share of exposures so unexplored options still generate data.
- Log exposure and outcome separately, so a click can be attributed to what was shown.
- Track diversity and coverage alongside accuracy, since narrowing is the first symptom.
- Audit generated content before it enters a corpus the model will retrieve from.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Pure exploitation | Best short-term metric | Narrows the data and hides alternatives | The metric is a proxy for long-term value |
| Randomised exploration slice | Unbiased reference data | Some users get worse results | Traffic is too small to spare a slice |
| Human review of generated content | Blocks bad data entering the corpus | Cost and latency | Volume is high and review becomes a rubber stamp |
| Synthetic data augmentation | Cheap coverage of rare cases | Amplifies model errors | The model's errors are the thing being measured |
Good for
Best short-term metric
Cost
Narrows the data and hides alternatives
Fails when
The metric is a proxy for long-term value
- The metric is a proxy for long-term value
- Traffic is too small to spare a slice
- Volume is high and review becomes a rubber stamp
- The model's errors are the thing being measured
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Pure exploitation is good for Best short-term metric, costs Narrows the data and hides alternatives, and fails when The metric is a proxy for long-term value. Randomised exploration slice is good for Unbiased reference data, costs Some users get worse results, and fails when Traffic is too small to spare a slice. Human review of generated content is good for Blocks bad data entering the corpus, costs Cost and latency, and fails when Volume is high and review becomes a rubber stamp. Synthetic data augmentation is good for Cheap coverage of rare cases, costs Amplifies model errors, and fails when The model's errors are the thing being measured
Failure modes
- Training on own output. The model is fine-tuned on answers it generated and its mistakes become ground truth. Debug: trace each training example to its source and count the share that is model-generated.
- Contaminated evaluation. The eval set contains model-written answers and scores rise without quality improving. Debug: check provenance of every eval item and hold out a human-authored set.
- Popularity collapse. Recommendations concentrate on a few items and coverage falls while engagement holds. Debug: chart distinct items shown per week against the exposure distribution.
In production
Monitor the share of model-generated data entering each corpus, and cap it. Keep the unbiased holdout out of every training and retrieval path, and re-check it after any change to the loop.
Common misconceptions
People think: Feedback loops are a recommendation-system problem.
Actually: Any system whose output becomes input has one, including retrieval corpora and coding agents that write files.
People think: More data from the model improves it.
Actually: It improves agreement with the model's own distribution, which is not the same as correctness and can reduce diversity.
Interview angle
Map the loop, identify the contaminated evidence and propose a randomised or human-authored holdout. Name the metric that decays first. Treating model output as free training data is the mistake to avoid.
Sources
- Sculley et al., Hidden Technical Debt in Machine Learning Systems — feedback loops and hidden dependencies in ML systems.
- Google, Rules of Machine Learning — measuring the metric you optimise and its proxies.