Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 18 of 22

Catastrophic forgetting

0 of 22 complete

Why it matters

Neural networks do not partition capacity by task. When training data shifts, the weights that produced old behaviour receive no gradient telling them to stay, so they drift toward whatever the new loss rewards. The result is a model that passes the new evaluation and fails on traffic it used to handle. The engineering problem is preserving behaviour you are not measuring.

Mental model

Forgetting is renovating one room and finding the plumbing inside the wall you removed: the change is local in intent and systemic in effect. The analogy breaks because you can inspect the old behaviour directly by running the base model on the same prompts.

How it works

Which bands move decides what is forgottenInteractive
Update scope
signalbypassfrozentrainable

Adapter · Step 1 of 5

Measure first

Score the base on a broad slice before training. Without that reference, a regression is indistinguishable from a change in preference.

Active layers

  • Base checkpoint
  • General slice

Hover a layer to read what it holds.

Next: Hold the base bands

Updated
adapter weights only
Frozen
embeddings · attention · FFN
Risk
weak adapter
Step 1/5 · Measure first

Figure. The same architecture under two update scopes. In the adapter arrangement the three base bands are frozen and only the adapter moves; in the full fine-tune every band is trainable and a narrow target loss reaches all of them. The general slice is measured before and after in both. The switch selects one complete stack at a time; mixing general data is a mitigation named in the steps, not a drawn band.

What to notice. Freezing is a property of the bands, not of the step order: the adapter arrangement cannot forget the base because those weights are never in the optimiser. The full arrangement is the failure case, and its sign is two curves separating, target up and general down. Neither drawing measures the size of the loss; it shows where the update lands and what that makes possible.

Text version

A switch labelled Update scope selects Adapter or Full fine-tune, each a complete vertical stack. Adapter: a base checkpoint feeds a frozen group holding embeddings, attention and feed-forward, and a dashed lane from the base attaches a single trainable adapter per layer; both the frozen forward path and the adapter reach the tuned model, which then reaches a general slice; a second dashed lane carries the base to the same slice as a baseline. Full fine-tune: the same base feeds a trainable group with the same three bands marked updated, whose logits enter a narrow target loss, then an optimiser update; the update writes the tuned model and returns to the group along a dash-dot lane labelled every weight, and a dashed lane anchors the baseline. Five steps per arrangement cover the baseline, the update scope, the update, the narrowing loss, and the two curves.

  1. Measure the base model on a broad slice before training, so you have a reference for what is being lost.
  2. Mix a share of general instruction data into the fine-tuning set, with the ratio as an explicit parameter.
  3. Prefer an adapter when the target change is behavioural, since freezing the base confines drift to the added weights.
  4. Lower the learning rate and shorten the schedule, because most forgetting happens in the first aggressive updates.
  5. Evaluate after every checkpoint on both the target and the general slice, and plot the two curves together.
  6. If forgetting persists, interleave batches from both distributions rather than training sequentially.

Trade-offs

ChoiceGood forCostFails when
Mix general dataCheap, effective, measurableDilutes the target signalThe target distribution is very narrow
Adapter onlyBase behaviour preservedCapacity limit on the changeYou need a new tokenizer or modality
Low learning rateLess driftSlower convergenceThe task needs a large shift
Rehearsal bufferDirectly targets old behaviourStorage and pipeline complexityYou cannot retain the old data
Trade-off mapInteractive

Good for

Cheap, effective, measurable

Cost

Dilutes the target signal

Fails when

The target distribution is very narrow

  1. The target distribution is very narrow
  2. You need a new tokenizer or modality
  3. The task needs a large shift
  4. You cannot retain the old data

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Mix general data is good for Cheap, effective, measurable, costs Dilutes the target signal, and fails when The target distribution is very narrow. Adapter only is good for Base behaviour preserved, costs Capacity limit on the change, and fails when You need a new tokenizer or modality. Low learning rate is good for Less drift, costs Slower convergence, and fails when The task needs a large shift. Rehearsal buffer is good for Directly targets old behaviour, costs Storage and pipeline complexity, and fails when You cannot retain the old data

Failure modes

  • Target-only evaluation. The tuned model passes and users report it cannot do the old task. Debug: run the base model's evaluation suite against the tuned model and diff per category.
  • Silent refusal drift. The model starts refusing benign requests after safety-heavy tuning. Debug: score a fixed benign prompt set before and after, and compare refusal rates.
  • Format lock-in. The model applies the new output format to unrelated requests. Debug: sample general prompts and check whether the tuned template appears unprompted.

In production

Keep a regression suite that runs on every checkpoint, not only at release. Alert on refusal rate and format compliance, which move before aggregate quality does.

Common misconceptions

People think: Forgetting only affects very small models.

Actually: It is a property of narrow sequential updates and appears at every scale, though the symptoms differ.

People think: A lower learning rate solves it.

Actually: It slows drift; only data mixing or parameter isolation addresses the cause.

Interview angle

Forgetting is interference from narrow gradients, so give the mitigations in cost order: data mixing, adapters, schedules. Say how you would detect it with a base-model reference suite. Weak answers name the phenomenon and stop, or claim PEFT eliminates it rather than confining it.

Sources