Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 2 of 38

Model serving

0 of 38 complete

Why it matters

A checkpoint on disk does nothing for a user. Something must hold it in device memory, decode tokens and keep answering while the process restarts, the model is replaced or a device is lost. That something is serving, and its hard problems are lifecycle problems: allocation happens once at boot, a failed warmup looks identical to a healthy process from outside, and a deploy that swaps weights mid-request returns an answer from neither version.

Mental model

Serving is the front of house and the engine is the kitchen: the host seats guests, takes orders and says when a dish has run out. The comparison breaks at capacity, because a restaurant can seat people and let them wait, while a serving replica that admits a request without cache room must either queue it or fail it.

How it works

Mechanism, step by stepInteractive
  1. 1Choose the engine from the traffic shape, then wrap it in a process supervisor that restarts on crash and reloads weights exactly once.
  2. 2Send a real generation through the model during startup and gate readiness on it, so a replica cannot receive traffic before kernels are compiled and the cache is allocated.
  3. 3Expose one HTTP contract for streaming and non-streaming calls, and propagate client disconnects so a cancelled request releases its blocks.
  4. 4Record the model revision, adapter set and engine version in every response header and in the startup log.
  5. 5On shutdown, stop accepting new requests, finish or cancel in-flight generations within a deadline, then release device memory.
  6. 6Keep liveness separate from readiness: a process that is alive but cannot generate should leave the pool, not restart in a loop.

Step 1 of 6

Choose the engine from the traffic shape, then wrap it in a process supervisor that restarts on crash and reloads weights exactly once.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Choose the engine from the traffic shape, then wrap it in a process supervisor that resta… 2. Send a real generation through the model during startup and gate readiness on it, so a re… 3. Expose one HTTP contract for streaming and non-streaming calls, and propagate client disc… 4. Record the model revision, adapter set and engine version in every response header and in… 5. On shutdown, stop accepting new requests, finish or cancel in-flight generations within a… 6. Keep liveness separate from readiness: a process that is alive but cannot generate should…

  1. Choose the engine from the traffic shape, then wrap it in a process supervisor that restarts on crash and reloads weights exactly once.
  2. Send a real generation through the model during startup and gate readiness on it, so a replica cannot receive traffic before kernels are compiled and the cache is allocated.
  3. Expose one HTTP contract for streaming and non-streaming calls, and propagate client disconnects so a cancelled request releases its blocks.
  4. Record the model revision, adapter set and engine version in every response header and in the startup log.
  5. On shutdown, stop accepting new requests, finish or cancel in-flight generations within a deadline, then release device memory.
  6. Keep liveness separate from readiness: a process that is alive but cannot generate should leave the pool, not restart in a loop.

Trade-offs

ChoiceGood forCostFails when
One process per devicePredictable memory, simple failure isolationIdle capacity when the model is smallTraffic is bursty and the model leaves headroom
Several models in one processPacking small models onto one cardFragmented memory and coupled crashesAny model needs the whole device
Serverless endpointNo cost when idleCold start dominated by weight loadingInteractive latency is the product
Managed inference endpointNo cluster operationsLess control over batching and versionsYou need custom kernels or tight cost control
Trade-off mapInteractive

Good for

Predictable memory, simple failure isolation

Cost

Idle capacity when the model is small

Fails when

Traffic is bursty and the model leaves headroom

  1. Traffic is bursty and the model leaves headroom
  2. Any model needs the whole device
  3. Interactive latency is the product
  4. You need custom kernels or tight cost control

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: One process per device is good for Predictable memory, simple failure isolation, costs Idle capacity when the model is small, and fails when Traffic is bursty and the model leaves headroom. Several models in one process is good for Packing small models onto one card, costs Fragmented memory and coupled crashes, and fails when Any model needs the whole device. Serverless endpoint is good for No cost when idle, costs Cold start dominated by weight loading, and fails when Interactive latency is the product. Managed inference endpoint is good for No cluster operations, costs Less control over batching and versions, and fails when You need custom kernels or tight cost control

Failure modes

  • Readiness before warmup. The pod reports healthy as soon as the port binds, so the first requests time out while weights load. Debug: make the readiness probe call a short generation and assert a non-empty completion.
  • Orphaned device memory. A crashed worker leaves its allocation held by a zombie parent and every restart fails with an out-of-memory error. Debug: compare device process ids with your supervisor's process tree after a forced kill.
  • Mixed revisions behind one service. Two replicas serve different checkpoints and answers change depending on which one the load balancer picked. Debug: return the revision in a response header and aggregate it by replica.

In production

Watch model load time, warmup failures and per-replica revision as first-class metrics, and alert when the ready pool shrinks. Rehearse a rolling restart under load, because that is when cache allocation and drain behaviour are actually exercised.

Common misconceptions

People think: Serving and the inference engine are the same thing.

Actually: The engine schedules and executes kernels; serving adds admission, streaming, health, versioning and lifecycle around it.

People think: More replicas always mean more throughput.

Actually: Each replica pays its own weight memory and cold start, and a shared upstream quota can cap total tokens regardless of replica count.

Interview angle

Start by separating the engine from the serving layer, then walk the lifecycle: load, warm, admit, stream, drain, version. Name the readiness probe as a correctness control and say what a cancelled request must release. Answers that describe an endpoint and a model name, and never mention a deploy or a device failure, miss the hard part.

Sources