Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 13 of 38

Serving engines

0 of 38 complete

Why it matters

Two engines running the same weights can differ substantially in tokens per second and in p95 latency, because one batches continuously while another runs fixed batches, or one supports a fused kernel for your dtype while the other falls back. The choice is constrained by hardware, architecture and the features your product needs, such as streaming, tool-call parsing or constrained decoding. A headline benchmark on a single synthetic request answers none of that.

Mental model

Engines are kitchens with different layouts: same ingredients, different stations and pass-through. A layout that plates one dish quickly can still stall on another. The metaphor breaks because you cannot taste the output to compare them; the only evidence is measured latency and quality on your own requests.

How it works

Mechanism, step by stepInteractive
  1. 1Write down the constraints before comparing: device vendor, architecture, dtype, maximum context and required features.
  2. 2Filter engines by hard requirements such as supported attention kernels, quantization formats and constrained decoding.
  3. 3Replay a sample of production prompts with the real input and output length distribution.
  4. 4Measure time-to-first-token, inter-token latency and goodput at several concurrency levels, not peak throughput alone.
  5. 5Check the operational surface: metrics, health, graceful shutdown, multi-model support and upgrade path.
  6. 6Pilot the leading candidate behind your own gateway so the engine can be swapped without touching application code.

Step 1 of 6

Write down the constraints before comparing: device vendor, architecture, dtype, maximum context and required features.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write down the constraints before comparing: device vendor, architecture, dtype, maximum … 2. Filter engines by hard requirements such as supported attention kernels, quantization for… 3. Replay a sample of production prompts with the real input and output length distribution. 4. Measure time-to-first-token, inter-token latency and goodput at several concurrency level… 5. Check the operational surface: metrics, health, graceful shutdown, multi-model support an… 6. Pilot the leading candidate behind your own gateway so the engine can be swapped without …

  1. Write down the constraints before comparing: device vendor, architecture, dtype, maximum context and required features.
  2. Filter engines by hard requirements such as supported attention kernels, quantization formats and constrained decoding.
  3. Replay a sample of production prompts with the real input and output length distribution.
  4. Measure time-to-first-token, inter-token latency and goodput at several concurrency levels, not peak throughput alone.
  5. Check the operational surface: metrics, health, graceful shutdown, multi-model support and upgrade path.
  6. Pilot the leading candidate behind your own gateway so the engine can be swapped without touching application code.

Trade-offs

ChoiceGood forCostFails when
General-purpose Python engineFast feature uptake, easy customisationOverhead and less predictable tailsThe latency budget is extremely tight
Vendor-optimised engineBest kernels on its hardwareNarrower model and hardware supportYou run mixed hardware or unusual architectures
CPU or edge engineRuns without a data-centre deviceMuch lower throughputConcurrency is more than a trickle
Managed engineNo operationsLimited tuning and visibilityYou need control over batching or cache
Trade-off mapInteractive

Good for

Fast feature uptake, easy customisation

Cost

Overhead and less predictable tails

Fails when

The latency budget is extremely tight

  1. The latency budget is extremely tight
  2. You run mixed hardware or unusual architectures
  3. Concurrency is more than a trickle
  4. You need control over batching or cache

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: General-purpose Python engine is good for Fast feature uptake, easy customisation, costs Overhead and less predictable tails, and fails when The latency budget is extremely tight. Vendor-optimised engine is good for Best kernels on its hardware, costs Narrower model and hardware support, and fails when You run mixed hardware or unusual architectures. CPU or edge engine is good for Runs without a data-centre device, costs Much lower throughput, and fails when Concurrency is more than a trickle. Managed engine is good for No operations, costs Limited tuning and visibility, and fails when You need control over batching or cache

Failure modes

  • Benchmark on the wrong distribution. Short synthetic prompts show a fast engine that collapses on long documents. Debug: replay the production length histogram and compare p95 at fixed concurrency.
  • Unsupported kernel path. A quantized model runs through a fallback and is slower than the float baseline. Debug: capture kernel names during a profile and confirm the fused path.
  • Feature gap discovered late. The engine cannot stream tool-call deltas or enforce a schema, so the application grows a parser it did not plan for. Debug: run your hardest request shape through the engine's API before committing.

In production

Keep the engine behind an internal interface and pin its version with the model. Track time-to-first-token and inter-token latency per engine version so an upgrade that changes scheduling is visible immediately.

Common misconceptions

People think: The engine with the highest tokens per second is the best choice.

Actually: Peak throughput is measured at saturation; interactive products are decided by tail latency at moderate concurrency.

People think: Engines are interchangeable because they expose a compatible API.

Actually: The surface matches; scheduling, cache behaviour, feature support and failure modes do not.

Interview angle

Start from constraints and traffic distribution, then eliminate engines on hard requirements before benchmarking. A strong answer reports time-to-first-token and inter-token latency at fixed concurrency. Weak answers quote a throughput number from a blog post.

Sources