Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 33 of 38

Feedback loops

0 of 38 complete

Why it matters

When a model's output becomes an input to future decisions, the data stops being an independent sample. A recommender that only shows popular items observes clicks only on popular items; an assistant that drafts tickets creates the corpus it will later retrieve from; an agent that writes files leaves artefacts that change its next run. The loop can improve the product and quietly destroy the evidence needed to tell whether it did.

Mental model

It is an echo in a room: the sound is real, but each reflection reinforces some frequencies and cancels others. The analogy breaks because an echo fades, while a data loop accumulates and can persist after the original cause is gone.

How it works

Mechanism, step by stepInteractive
  1. 1Map where model output enters future inputs: training data, retrieval corpus, ranking features, prompt examples.
  2. 2Keep a holdout that is never influenced by model output, and use it for evaluation.
  3. 3Randomise a small share of exposures so unexplored options still generate data.
  4. 4Log exposure and outcome separately, so a click can be attributed to what was shown.
  5. 5Track diversity and coverage alongside accuracy, since narrowing is the first symptom.
  6. 6Audit generated content before it enters a corpus the model will retrieve from.

Step 1 of 6

Map where model output enters future inputs: training data, retrieval corpus, ranking features, prompt examples.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Map where model output enters future inputs: training data, retrieval corpus, ranking fea… 2. Keep a holdout that is never influenced by model output, and use it for evaluation. 3. Randomise a small share of exposures so unexplored options still generate data. 4. Log exposure and outcome separately, so a click can be attributed to what was shown. 5. Track diversity and coverage alongside accuracy, since narrowing is the first symptom. 6. Audit generated content before it enters a corpus the model will retrieve from.

  1. Map where model output enters future inputs: training data, retrieval corpus, ranking features, prompt examples.
  2. Keep a holdout that is never influenced by model output, and use it for evaluation.
  3. Randomise a small share of exposures so unexplored options still generate data.
  4. Log exposure and outcome separately, so a click can be attributed to what was shown.
  5. Track diversity and coverage alongside accuracy, since narrowing is the first symptom.
  6. Audit generated content before it enters a corpus the model will retrieve from.

Trade-offs

ChoiceGood forCostFails when
Pure exploitationBest short-term metricNarrows the data and hides alternativesThe metric is a proxy for long-term value
Randomised exploration sliceUnbiased reference dataSome users get worse resultsTraffic is too small to spare a slice
Human review of generated contentBlocks bad data entering the corpusCost and latencyVolume is high and review becomes a rubber stamp
Synthetic data augmentationCheap coverage of rare casesAmplifies model errorsThe model's errors are the thing being measured
Trade-off mapInteractive

Good for

Best short-term metric

Cost

Narrows the data and hides alternatives

Fails when

The metric is a proxy for long-term value

  1. The metric is a proxy for long-term value
  2. Traffic is too small to spare a slice
  3. Volume is high and review becomes a rubber stamp
  4. The model's errors are the thing being measured

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Pure exploitation is good for Best short-term metric, costs Narrows the data and hides alternatives, and fails when The metric is a proxy for long-term value. Randomised exploration slice is good for Unbiased reference data, costs Some users get worse results, and fails when Traffic is too small to spare a slice. Human review of generated content is good for Blocks bad data entering the corpus, costs Cost and latency, and fails when Volume is high and review becomes a rubber stamp. Synthetic data augmentation is good for Cheap coverage of rare cases, costs Amplifies model errors, and fails when The model's errors are the thing being measured

Failure modes

  • Training on own output. The model is fine-tuned on answers it generated and its mistakes become ground truth. Debug: trace each training example to its source and count the share that is model-generated.
  • Contaminated evaluation. The eval set contains model-written answers and scores rise without quality improving. Debug: check provenance of every eval item and hold out a human-authored set.
  • Popularity collapse. Recommendations concentrate on a few items and coverage falls while engagement holds. Debug: chart distinct items shown per week against the exposure distribution.

In production

Monitor the share of model-generated data entering each corpus, and cap it. Keep the unbiased holdout out of every training and retrieval path, and re-check it after any change to the loop.

Common misconceptions

People think: Feedback loops are a recommendation-system problem.

Actually: Any system whose output becomes input has one, including retrieval corpora and coding agents that write files.

People think: More data from the model improves it.

Actually: It improves agreement with the model's own distribution, which is not the same as correctness and can reduce diversity.

Interview angle

Map the loop, identify the contaminated evidence and propose a randomised or human-authored holdout. Name the metric that decays first. Treating model output as free training data is the mistake to avoid.

Sources