Why it matters
Once a team has one adapter per task, the next request is a combination: a tone adapter plus a domain adapter plus a format adapter. Training every combination does not scale, so the alternatives are to merge weights or to combine outputs at inference. Both look additive and neither is, because each adapter was trained against a frozen base and in the absence of the others.
Mental model
Composing adapters is mixing paints: two colours can combine predictably, but three often make mud, and the order you add them changes the result. The metaphor breaks because paint mixes once and stays mixed, while adapters can be routed per request and each combination can be evaluated on its own before anyone sees it.
How it works
Merge · Step 1 of 3
Trained in isolation
Each adapter was trained against the frozen base with no knowledge of the others. Individual scores say nothing about a request that needs two behaviours.
Active layers
- Trained in isolation
- Tone adapter
- Domain adapter
- Format adapter
Hover a layer to read what it holds.
Next: Merge the weights
- Composition
- weight-space merge
- Serving
- no extra cost
- Risk
- interference · scale drift
Figure. Two ways to compose adapters trained separately against the same frozen base, selected with the switch. Merge combines the three updates once in weight space; Route selects one adapter per request. They are alternatives, not consecutive stages.
What to notice. Weight merging has no serving overhead but can suffer interference and scale drift, and the composed model is a single blend. Routing avoids interference but adds an argmax router and can send a request to the wrong adapter. Component scores do not predict the composition: both paths need their own evaluation on requests that require more than one behaviour.
Text version
A switch labelled Composition selects one of two complete stacks. Both start with a request and the frozen base, then list three adapters trained in isolation: tone, domain and format. Merge shows a separate right-hand lane from each adapter into a weight sum, which produces one merged model. Route shows a single lane from the adapters group to an argmax router, which selects one adapter and produces a routed output. Each stack has its own steps; no edge joins the two paths.
- Classify the adapters by what they change, such as knowledge, format, tone or reasoning, since same-kind adapters interfere more.
- Measure each adapter alone, then every pair you intend to serve, on a shared held-out set that covers both behaviours.
- Try weight-space merging first when the adapters touch disjoint modules or when one is much smaller.
- Use activation-level composition when behaviours conflict, so a router can choose per request instead of averaging.
- Watch for scale drift, because merging rescales updates and a naive average can shrink or amplify the base model's behaviour.
- Version the composed artefact as its own model, with the component revisions recorded.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Weight averaging | No serving overhead | Interference and scale drift | Adapters were trained on different data regimes |
| Task arithmetic | Composable and cheap to try | Sign flips and magnitude sensitivity | Behaviours are entangled |
| Activation mixing | Learned weighting per input | Extra parameters and training | You cannot train a router |
| Routing by request | No interference | Router errors and per-request cost | Requests need several behaviours at once |
Good for
No serving overhead
Cost
Interference and scale drift
Fails when
Adapters were trained on different data regimes
- Adapters were trained on different data regimes
- Behaviours are entangled
- You cannot train a router
- Requests need several behaviours at once
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Weight averaging is good for No serving overhead, costs Interference and scale drift, and fails when Adapters were trained on different data regimes. Task arithmetic is good for Composable and cheap to try, costs Sign flips and magnitude sensitivity, and fails when Behaviours are entangled. Activation mixing is good for Learned weighting per input, costs Extra parameters and training, and fails when You cannot train a router. Routing by request is good for No interference, costs Router errors and per-request cost, and fails when Requests need several behaviours at once
Failure modes
- Silent interference. Each adapter passes alone and the pair regresses on both tasks. Debug: evaluate the pair on a set that requires both behaviours and compare against each component.
- Scale drift after averaging. The merged model is flatter and less decisive than either component. Debug: compare output entropy and refusal rate against each adapter.
- Router defaulting. The router sends nearly everything to one adapter and the other is dead weight. Debug: log the routing distribution over a real traffic sample.
In production
Treat each composition as a release with its own evaluation record. Monitor the routing distribution and the per-behaviour success rate, since a composition can degrade without any component changing.
Common misconceptions
People think: Adapters compose additively.
Actually: They were trained independently against a frozen base, so their updates interact in ways neither was optimised for.
People think: If each adapter passes, the combination is safe.
Actually: Passing alone says nothing about a request that needs two behaviours, which is exactly what composition is for.
Interview angle
Distinguish weight-space from activation-space composition and say why interference is the central risk. Name the evaluation you would run, a pair set that requires both behaviours. Saying you would average the adapters without mentioning scale or interference is the shallow answer.
Sources
- Chronopoulou et al., AdapterFusion: Non-Destructive Task Composition for Transfer Learning — combining adapter outputs with a learned weighting.
- Yadav et al., TIES-Merging: Resolving Interference When Merging Models — sign conflicts and interference in weight-space merging.
- Wortsman et al., Model Soups: Averaging Weights of Multiple Fine-tuned Models — weight averaging as a baseline for composition.