Why it matters
Every product decision about an LLM eventually becomes an inference decision. A chat interface needs a fast first token. A batch enrichment job needs high throughput and does not care about latency. A voice agent needs both, plus predictable tail behaviour. The same model, served differently, produces very different cost and user experience.
Inference is also where most of the bill is. Training is a large one-off; serving is a recurring cost per request, and it scales with prompt length, output length and concurrency. Understanding the two phases explains why long prompts are cheap per token compared with long outputs, and why a cached prefix can cut cost without changing the model.
Mental model
Reading a brief and then writing a reply by hand. Reading is parallel: you can absorb the whole document at once, and the work scales with its length. Writing is serial: you produce one word at a time, and each word depends on everything you have written so far.
The analogy breaks because a serving system does not work on one request at a time. It interleaves many requests, so the "writer" is switching between dozens of documents at each word. That interleaving is what makes throughput possible and what makes latency unpredictable under load.
How it works
- 1Queue: the request waits for a slot, or for a batch to be assembled.
- 2Prefill: the prompt tokens run through the model in parallel, filling the KV cache.
- 3First token: logits from the final prompt position are sampled and returned.
- 4Decode: each new token requires one forward pass that reads the weights and the KV cache.
- 5Finish: generation stops on an end token, a stop sequence, or the output cap.
Prefill is compute-bound. The whole prompt is a matrix of tokens, so the GPU can process it in large parallel operations. This is why prompt tokens are generally cheaper than output tokens and why a long prompt produces a noticeable delay before the first token.
Decode is memory-bandwidth-bound. To emit one token, the model must read its weights and the cached keys and values for every previous token. The arithmetic per byte is low, so the wall-clock cost is dominated by moving data, not by multiplying it. Larger batches amortise the weight read across requests, which is the entire economic argument for continuous batching: the server adds new requests to the running batch at every decode step instead of waiting for a batch to finish.
The KV cache is the memory that makes this possible and the resource that limits it. Every request holds the keys and values for its prompt and generated tokens. When cache memory runs out, the server either rejects work, queues it, or preempts a request and recomputes later. Paged attention manages this cache in fixed-size blocks so memory can be allocated and freed without large contiguous reservations.
Quick facts
- Prefill
- Parallel, compute-bound
- Decode
- Serial, bandwidth-bound
- TTFT
- Queue + prefill
- ITL
- Per-token decode time
- Batching
- Throughput up, latency up
- Cache
- KV per request, grows with tokens
The latency identity
For a request that generates tokens, end-to-end latency decomposes into:
Time to first token (TTFT) is plus network overhead. Inter-token latency (ITL) is , and the full request is roughly . A change that improves one can hurt the other: adding requests to a batch shrinks the weight-read cost per token but lengthens the queue and slows each step.
Throughput follows Little's law. With requests in flight and average latency , completed requests per second is roughly . This is why a service can report high tokens per second while individual users wait: tokens per second is an aggregate, not a per-user guarantee.
Measuring what users feel
import time
def timed_stream(stream):
"""stream yields text deltas; timings use a monotonic clock."""
start = time.perf_counter()
first = None
last = start
chunks = 0
for delta in stream:
now = time.perf_counter()
if first is None:
first = now
last = now
chunks += 1
return {
"ttft_s": None if first is None else first - start,
"total_s": last - start,
"chunks": chunks,
}Measure at the client, not only at the server. Client TTFT includes network and proxy buffering, and a service that streams from the model but buffers at the edge looks identical to a slow service to the user. Record TTFT, ITL and total duration as separate histograms, by model and by prompt size bucket.
Text version
A client request passes a gateway that authenticates and routes it, a scheduler batches it, the model prefills and decodes, and tokens stream back.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Hosted API, streaming | Fast start, no operations | Per-token price, shared capacity | You need strict data residency or guaranteed capacity |
| Dedicated hosted endpoint | Predictable latency, isolation | Idle capacity you pay for | Traffic is bursty and mostly idle |
| Self-hosted with continuous batching | High utilisation, custom models | GPU fleet, serving expertise | Load is low or highly variable |
| Batch/offline inference | Enrichment, evals, backfills | Hours of delay | The result is needed interactively |
Good for
Fast start, no operations
Cost
Per-token price, shared capacity
Fails when
You need strict data residency or guaranteed capacity
- You need strict data residency or guaranteed capacity
- Traffic is bursty and mostly idle
- Load is low or highly variable
- The result is needed interactively
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Hosted API, streaming is good for Fast start, no operations, costs Per-token price, shared capacity, and fails when You need strict data residency or guaranteed capacity. Dedicated hosted endpoint is good for Predictable latency, isolation, costs Idle capacity you pay for, and fails when Traffic is bursty and mostly idle. Self-hosted with continuous batching is good for High utilisation, custom models, costs GPU fleet, serving expertise, and fails when Load is low or highly variable. Batch/offline inference is good for Enrichment, evals, backfills, costs Hours of delay, and fails when The result is needed interactively
Failure modes
TTFT spikes with flat GPU utilisation. The queue is the problem, not the model. Debug by instrumenting queue time separately from prefill time; if queue time dominates, the admission control or batch size is wrong for the arrival rate.
Throughput collapses as concurrency rises. KV-cache pressure causes preemption or recomputation. Debug by tracking active sequences, cache block usage and preemption counts. If cache usage is pinned at the limit, cap concurrency or shorten the context before adding replicas.
Outputs end mid-sentence. The output cap or a stop sequence fired. Debug the finish reason and the configured token limit. A truncated JSON object and a deliberately short answer look the same in the text but need different fixes.
A new model version is slower at the same token count. Tokenisers differ, and a prompt that was 800 tokens on the old model may be 950 on the new one. Debug by re-counting tokens with the matching tokeniser rather than assuming the prompt is unchanged.
In production
Autoscale on queue depth and TTFT, not on GPU utilisation; utilisation looks healthy while requests wait. Cache stable prefixes — system prompts, tool definitions, shared documents — and verify the cache is actually hit. Cap output tokens per endpoint, because unbounded generation is the most common cause of a latency incident. Track cost per successful request, not per token, so retries and abandoned streams count. For user-facing chat, treat p95 TTFT as the primary SLO and ITL as the secondary one.
Common misconceptions
People think: High GPU utilisation means inference is healthy.
Actually: A saturated GPU can be stalled on memory bandwidth or serving a huge batch while each user waits. Measure queue time, TTFT and ITL.
People think: Tokens per second is the number a user experiences.
Actually: Aggregate tokens per second describes the fleet. A user experiences TTFT and the gaps between tokens under their own concurrency.
Interview angle
Expect a question that asks you to explain a latency spike or size a deployment. The strong answer separates queueing, prefill and decode, names the metric for each, and states which lever changes which term. The weak answer says "add more GPUs". Work through the LLM fundamentals questions and the production interview bank.
Sources
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention — paged KV cache and its effect on serving throughput.
- Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models — iteration-level continuous batching.
- NVIDIA, Mastering LLM Techniques: Inference Optimization — prefill/decode, batching and memory behaviour.
- Databricks, LLM Inference Performance Engineering: Best Practices — latency and throughput metrics in practice.
Related
- LLM — what is being served.
- Tokens — the unit that drives prefill and decode cost.
- KV cache — the mechanism behind decode cost.
- Prefill and decode — the two phases at the model level.
- Latency and TTFT — operationalising the metrics.