Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 5 of 28

Inference

0 of 28 complete

Why it matters

Every product decision about an LLM eventually becomes an inference decision. A chat interface needs a fast first token. A batch enrichment job needs high throughput and does not care about latency. A voice agent needs both, plus predictable tail behaviour. The same model, served differently, produces very different cost and user experience.

Inference is also where most of the bill is. Training is a large one-off; serving is a recurring cost per request, and it scales with prompt length, output length and concurrency. Understanding the two phases explains why long prompts are cheap per token compared with long outputs, and why a cached prefix can cut cost without changing the model.

Mental model

Reading a brief and then writing a reply by hand. Reading is parallel: you can absorb the whole document at once, and the work scales with its length. Writing is serial: you produce one word at a time, and each word depends on everything you have written so far.

The analogy breaks because a serving system does not work on one request at a time. It interleaves many requests, so the "writer" is switching between dozens of documents at each word. That interleaving is what makes throughput possible and what makes latency unpredictable under load.

How it works

  1. 1Queue: the request waits for a slot, or for a batch to be assembled.
  2. 2Prefill: the prompt tokens run through the model in parallel, filling the KV cache.
  3. 3First token: logits from the final prompt position are sampled and returned.
  4. 4Decode: each new token requires one forward pass that reads the weights and the KV cache.
  5. 5Finish: generation stops on an end token, a stop sequence, or the output cap.

Prefill is compute-bound. The whole prompt is a matrix of tokens, so the GPU can process it in large parallel operations. This is why prompt tokens are generally cheaper than output tokens and why a long prompt produces a noticeable delay before the first token.

Decode is memory-bandwidth-bound. To emit one token, the model must read its weights and the cached keys and values for every previous token. The arithmetic per byte is low, so the wall-clock cost is dominated by moving data, not by multiplying it. Larger batches amortise the weight read across requests, which is the entire economic argument for continuous batching: the server adds new requests to the running batch at every decode step instead of waiting for a batch to finish.

The KV cache is the memory that makes this possible and the resource that limits it. Every request holds the keys and values for its prompt and generated tokens. When cache memory runs out, the server either rejects work, queues it, or preempts a request and recomputes later. Paged attention manages this cache in fixed-size blocks so memory can be allocated and freed without large contiguous reservations.

Quick facts

Prefill
Parallel, compute-bound
Decode
Serial, bandwidth-bound
TTFT
Queue + prefill
ITL
Per-token decode time
Batching
Throughput up, latency up
Cache
KV per request, grows with tokens

The latency identity

For a request that generates NN tokens, end-to-end latency decomposes into:

L≈Lqueue+Lprefill+(N−1)⋅LdecodeL \approx L_{\text{queue}} + L_{\text{prefill}} + (N-1)\cdot L_{\text{decode}}

Time to first token (TTFT) is Lqueue+LprefillL_{\text{queue}} + L_{\text{prefill}} plus network overhead. Inter-token latency (ITL) is LdecodeL_{\text{decode}}, and the full request is roughly TTFT+(N−1)⋅ITL\text{TTFT} + (N-1)\cdot\text{ITL}. A change that improves one can hurt the other: adding requests to a batch shrinks the weight-read cost per token but lengthens the queue and slows each step.

Throughput follows Little's law. With CC requests in flight and average latency LL, completed requests per second is roughly C/LC/L. This is why a service can report high tokens per second while individual users wait: tokens per second is an aggregate, not a per-user guarantee.

Measuring what users feel

import time
 
def timed_stream(stream):
    """stream yields text deltas; timings use a monotonic clock."""
    start = time.perf_counter()
    first = None
    last = start
    chunks = 0
 
    for delta in stream:
        now = time.perf_counter()
        if first is None:
            first = now
        last = now
        chunks += 1
 
    return {
        "ttft_s": None if first is None else first - start,
        "total_s": last - start,
        "chunks": chunks,
    }

Measure at the client, not only at the server. Client TTFT includes network and proxy buffering, and a service that streams from the model but buffers at the edge looks identical to a slow service to the user. Record TTFT, ITL and total duration as separate histograms, by model and by prompt size bucket.

ClientGatewayQueueModelStreamResponse
Step 1/5 · Client sends a request
Figure. A token's latency is the sum of every hop before the first token, plus decode.
Text version

A client request passes a gateway that authenticates and routes it, a scheduler batches it, the model prefills and decodes, and tokens stream back.

Trade-offs

ChoiceGood forCostFails when
Hosted API, streamingFast start, no operationsPer-token price, shared capacityYou need strict data residency or guaranteed capacity
Dedicated hosted endpointPredictable latency, isolationIdle capacity you pay forTraffic is bursty and mostly idle
Self-hosted with continuous batchingHigh utilisation, custom modelsGPU fleet, serving expertiseLoad is low or highly variable
Batch/offline inferenceEnrichment, evals, backfillsHours of delayThe result is needed interactively
Trade-off mapInteractive

Good for

Fast start, no operations

Cost

Per-token price, shared capacity

Fails when

You need strict data residency or guaranteed capacity

  1. You need strict data residency or guaranteed capacity
  2. Traffic is bursty and mostly idle
  3. Load is low or highly variable
  4. The result is needed interactively

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Hosted API, streaming is good for Fast start, no operations, costs Per-token price, shared capacity, and fails when You need strict data residency or guaranteed capacity. Dedicated hosted endpoint is good for Predictable latency, isolation, costs Idle capacity you pay for, and fails when Traffic is bursty and mostly idle. Self-hosted with continuous batching is good for High utilisation, custom models, costs GPU fleet, serving expertise, and fails when Load is low or highly variable. Batch/offline inference is good for Enrichment, evals, backfills, costs Hours of delay, and fails when The result is needed interactively

Failure modes

TTFT spikes with flat GPU utilisation. The queue is the problem, not the model. Debug by instrumenting queue time separately from prefill time; if queue time dominates, the admission control or batch size is wrong for the arrival rate.

Throughput collapses as concurrency rises. KV-cache pressure causes preemption or recomputation. Debug by tracking active sequences, cache block usage and preemption counts. If cache usage is pinned at the limit, cap concurrency or shorten the context before adding replicas.

Outputs end mid-sentence. The output cap or a stop sequence fired. Debug the finish reason and the configured token limit. A truncated JSON object and a deliberately short answer look the same in the text but need different fixes.

A new model version is slower at the same token count. Tokenisers differ, and a prompt that was 800 tokens on the old model may be 950 on the new one. Debug by re-counting tokens with the matching tokeniser rather than assuming the prompt is unchanged.

In production

Autoscale on queue depth and TTFT, not on GPU utilisation; utilisation looks healthy while requests wait. Cache stable prefixes — system prompts, tool definitions, shared documents — and verify the cache is actually hit. Cap output tokens per endpoint, because unbounded generation is the most common cause of a latency incident. Track cost per successful request, not per token, so retries and abandoned streams count. For user-facing chat, treat p95 TTFT as the primary SLO and ITL as the secondary one.

Common misconceptions

People think: High GPU utilisation means inference is healthy.

Actually: A saturated GPU can be stalled on memory bandwidth or serving a huge batch while each user waits. Measure queue time, TTFT and ITL.

People think: Tokens per second is the number a user experiences.

Actually: Aggregate tokens per second describes the fleet. A user experiences TTFT and the gaps between tokens under their own concurrency.

Interview angle

Expect a question that asks you to explain a latency spike or size a deployment. The strong answer separates queueing, prefill and decode, names the metric for each, and states which lever changes which term. The weak answer says "add more GPUs". Work through the LLM fundamentals questions and the production interview bank.

Sources