Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 21 of 22

Adapter serving

0 of 22 complete

Why it matters

Serving a hundred fine-tunes as a hundred models multiplies memory by a hundred. Adapter serving turns that into one base plus a cache of small deltas, which is a scheduling problem rather than a training one. The constraints are memory for the cache, batching efficiency when requests use different adapters, and the cold-start latency of loading an adapter that has not been used recently.

Mental model

Adapter serving is a library with one reference copy and a shelf of bookmarks: readers share the book and each brings their own marker. The analogy breaks because a bookmark cannot change the text, while an adapter changes every forward pass it touches.

How it works

One base at boot, one adapter per requestInteractive
signalbypassloop

Step 1 of 6

Load the base once

The base is read once at start-up and shared. It is a side input to the apply band, not a step in the request spine.

Active layers

  • Base weights

Hover a layer to read what it holds.

Next: Resolve the adapter

Base
one copy, at boot
Adapter
loaded on miss
Cache
bounded LRU
Step 1/6 · Load the base once

Figure. A two-step load: the base is read once and stays resident, while the adapter delta is loaded only on a cache miss and applied per request. The steady-state spine is request, key, cache, apply, response and skips the load group; a miss detours through store and verify, then rejoins apply. The base is a side input to apply, not a stage of the request. The store and its versioning are illustrative; batching behaviour is stated in the steps.

What to notice. The request path and the load path are separate: a hit never touches the adapter store, and a miss detours through the load group before rejoining the same apply band. That is why cache hit rate and adapter cold-start, not raw model size, decide the deployment's tail latency. A base revision change that invalidates stored adapters turns every request into the detour.

Text version

A vertical stack. A resident base weight band and a request both feed the request path: the request goes to a routing key, an adapter cache, an apply-adapter band and the response. A solid lane from the cache to apply skips the load group, while a dashed lane enters a miss group holding the adapter store and a verify-and-load band; the loaded delta rejoins apply and populates the cache along a dash-dot lane. A dashed lane carries the resident base directly to apply. Six steps cover the one-time base load, the routing key, a cache hit, a miss and verification, rejoining and cache population, and the throughput failure.

  1. Decide the routing key: tenant, task, locale or an explicit adapter id in the request.
  2. Load adapters lazily into a bounded cache and evict by least recent use, with a cap on total adapter memory.
  3. Group requests by adapter before batching, or use a serving stack that supports mixed-adapter batches.
  4. Measure cold-start latency for an adapter load, and keep hot adapters pinned for your busiest tenants.
  5. Version adapters by base model revision, and reject requests that name an incompatible pair.
  6. Monitor per-adapter latency and error rate, since one bad adapter can degrade a shared deployment.

Trade-offs

ChoiceGood forCostFails when
Merged per tenantNo routing or cacheOne model copy per tenantTenants are many and small
Dynamic adapter loadingOne base, many variantsCache misses and batching limitsTraffic is spread thin across adapters
Pinned adaptersPredictable latencyMemory reserved for idle variantsYou have a long tail of tenants
Single shared adapterSimplest operationNo per-tenant behaviourTenants need distinct behaviour
Trade-off mapInteractive

Good for

No routing or cache

Cost

One model copy per tenant

Fails when

Tenants are many and small

  1. Tenants are many and small
  2. Traffic is spread thin across adapters
  3. You have a long tail of tenants
  4. Tenants need distinct behaviour

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Merged per tenant is good for No routing or cache, costs One model copy per tenant, and fails when Tenants are many and small. Dynamic adapter loading is good for One base, many variants, costs Cache misses and batching limits, and fails when Traffic is spread thin across adapters. Pinned adapters is good for Predictable latency, costs Memory reserved for idle variants, and fails when You have a long tail of tenants. Single shared adapter is good for Simplest operation, costs No per-tenant behaviour, and fails when Tenants need distinct behaviour

Failure modes

  • Cache thrash. Latency spikes as adapters evict each other under mixed traffic. Debug: log cache hit rate and adapter load counts over an hour, and check whether the working set exceeds the cache.
  • Batch splitting. Throughput collapses because every request needs its own batch. Debug: measure mean batch size per adapter and compare with the single-adapter baseline.
  • Base and adapter mismatch. Requests return degraded output after a base model upgrade. Debug: record the base revision with each adapter and assert compatibility at load time.

In production

Alert on adapter cache hit rate and cold-start p99, not only on aggregate latency. Cap the number of resident adapters so a traffic spike cannot exhaust memory.

Common misconceptions

People think: Adapter serving is free compared with separate models.

Actually: It trades memory for scheduling complexity, and batching efficiency usually falls.

People think: Any adapter works with any base of the same family.

Actually: Adapters are trained against a specific checkpoint; a different revision changes the result.

Interview angle

Describe the memory arithmetic and the batching problem, then the cache policy. A strong answer names the long-tail tenant who keeps getting evicted. Weak answers say you load adapters on demand without discussing batching or cold starts.

Sources