Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 16 of 38

Latency & TTFT

0 of 38 complete

Why it matters

A single "latency" number hides the thing users actually feel. A chat interface that shows the first token in 400 ms and then dribbles the rest at 40 ms per token feels responsive. One that waits 4 seconds and then prints instantly feels broken, even though the total time is the same. Streaming turned that difference into a product requirement.

The split also matters operationally because the two phases fail differently. Time to first token (TTFT) is a queueing and prefill problem: how many requests are ahead of you, and how many prompt tokens must be processed before any output exists. Inter-token latency (ITL) is a decode problem: each generated token requires a forward pass that reads the whole model and the KV cache, so it is bound by memory bandwidth more than by arithmetic. Fixing one does nothing for the other.

Mental model

A restaurant with one kitchen. TTFT is how long you wait for the first plate; ITL is the pace at which the rest of the courses arrive. A busy kitchen delays the first plate because your order is behind others in the queue; a slow kitchen delays every course because each one is cooked to order.

The analogy breaks at parallelism. A kitchen can cook several dishes at once and can start plating before everything is done, which is roughly continuous batching. But a language model produces tokens strictly in sequence per request, so it cannot start the tenth token before the ninth. Decode is sequential within a request: the harness cannot start token ten before token nine. Speculative decoding proposes several tokens per step and verifies them together, but it still commits them in order; real parallelism comes from batching across requests, not from splitting one sequence.

How it works

Mechanism, step by stepInteractive
  1. 1Request arrives and is tokenised. Prompt length in tokens, not characters, determines the prefill cost.
  2. 2Scheduler admits the request. If every slot is occupied, it waits. Queueing delay is part of TTFT and is the component that collapses under load.
  3. 3Prefill runs. The model processes all prompt tokens in parallel, writing a KV cache entry for each. Cost scales with prompt length and grows superlinearly for very long prompts because attention compares every token with every other.
  4. 4First token is sampled and emitted. The clock from request start to this moment is TTFT.
  5. 5Decode loop. One forward pass per token, each reading the KV cache. The gap between consecutive tokens is ITL; its reciprocal is tokens per second. TPOT (time per output token) is the mean ITL, so it can hide a single mid-stream stall — report both when the tail matters.
  6. 6Stream terminates. Either a stop sequence, the output cap, or a length limit. Total time is the sum of the phases.

Step 1 of 6

Request arrives and is tokenised. Prompt length in tokens, not characters, determines the prefill cost.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Request arrives and is tokenised. Prompt length in tokens, not characters, determines the… 2. Scheduler admits the request. If every slot is occupied, it waits. Queueing delay is part… 3. Prefill runs. The model processes all prompt tokens in parallel, writing a KV cache entry… 4. First token is sampled and emitted. The clock from request start to this moment is TTFT. 5. Decode loop. One forward pass per token, each reading the KV cache. The gap between conse… 6. Stream terminates. Either a stop sequence, the output cap, or a length limit. Total time …

  1. Request arrives and is tokenised. Prompt length in tokens, not characters, determines the prefill cost.
  2. Scheduler admits the request. If every slot is occupied, it waits. Queueing delay is part of TTFT and is the component that collapses under load.
  3. Prefill runs. The model processes all prompt tokens in parallel, writing a KV cache entry for each. Cost scales with prompt length and grows superlinearly for very long prompts because attention compares every token with every other.
  4. First token is sampled and emitted. The clock from request start to this moment is TTFT.
  5. Decode loop. One forward pass per token, each reading the KV cache. The gap between consecutive tokens is ITL; its reciprocal is tokens per second. TPOT (time per output token) is the mean ITL, so it can hide a single mid-stream stall — report both when the tail matters.
  6. Stream terminates. Either a stop sequence, the output cap, or a length limit. Total time is the sum of the phases.

Sizing concurrency uses Little's law: the average number of in-flight requests LL equals arrival rate λ\lambda times time in system WW. At 12 requests per second and 1.5 s per request, you need about 18 concurrent request slots before queueing begins. Above that, wait time grows faster than load, because arrivals and service times both vary.

Measuring it honestly

Measure at two boundaries. Server-side, you get the provider's or your own serving stack's view. Client-side, you get what the user experienced, including TLS, network and any proxy in between. When they disagree, the gap is usually buffering.

import json
import time
 
import httpx
 
 
def stream_timings(url: str, payload: dict, headers: dict) -> dict:
    """Illustrative: OpenAI-compatible SSE shape. Other protocols differ."""
    t0 = time.perf_counter()
    first = None
    stamps = []
 
    with httpx.stream("POST", url, json=payload, headers=headers, timeout=60.0) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line.startswith("data:"):
                continue
            body = line[5:].strip()
            if body == "[DONE]":
                break
            delta = json.loads(body)["choices"][0]["delta"]
            if delta.get("content"):
                now = time.perf_counter()
                first = first or now
                stamps.append(now)
 
    gaps = sorted(b - a for a, b in zip(stamps, stamps[1:]))
    return {
        "ttft_ms": round((first - t0) * 1000, 1) if first else None,
        "itl_p50_ms": round(gaps[len(gaps) // 2] * 1000, 1) if gaps else None,
        "total_ms": round((stamps[-1] - t0) * 1000, 1) if stamps else None,
        "chunks": len(stamps),
    }

Record these per request with the route, model version, prompt token count, cached-token count and finish reason. Without those dimensions you can see that a percentile moved but not why.

Latency quantiles under loadInteractive
concurrency 8concurrency 11000 ms SLO
p50
352ms
p95
677ms
p99
888ms

At concurrency 8, p99 is 888 ms — inside the 1000 ms SLO. In this model p99 passes the SLO at concurrency 12, p95 at 23. The crossing follows from the constants, not from a measured service.

Illustrative, not a benchmark

Model: log-normal quantiles, L(p) = p50 · exp(σ · z(p)), with p50 = 320 + 4.5(c−1) ms and σ = 0.337 + 0.0088(c−1).

Assumes one queue, no retries and stable request sizes. Real tails also move with prompt length, cache hits and cold replicas.

Figure. Modelled latency quantiles at concurrency 1–32. The median rises about 44%; p95 and p99 climb faster, and in this model the tail passes a 1000 ms SLO.

What to notice. An average would hide this: from concurrency 1 to 32, p50 moves 44% while p99 moves 2.7×. The crossing points follow from the formula, not from a measured service, so treat them as a demonstration of the shape, not a capacity estimate.

Text version

A curve plotting latency in milliseconds against percentile from 0 to 100. A slider sets concurrency from 1 to 32. At concurrency 1 the curve reads p50 320 ms, p95 557 ms and p99 701 ms. At concurrency 32 it reads p50 460 ms, p95 1253 ms and p99 1899 ms. A dashed line marks the 1000 ms SLO. A ghost curve shows concurrency 1 for comparison, and the region above the SLO is shaded only when the current p99 exceeds it. The load model is invented and labelled illustrative; the crossing points are properties of that model, not capacity numbers.

Trade-offs

LeverEffect on TTFTEffect on ITL / totalFails when
Shorter promptLarge improvementNeutralThe task genuinely needs the context
Prefix / prompt cachingLarge on repeated prefixesNeutralPrefixes differ per user or request
Smaller or distilled modelImprovementImprovementQuality floor is already tight
Speculative decodingNeutral to slightly worseLarge improvementDraft model acceptance rate is low
Larger batches, higher utilisationWorse tailsWorse tailsYou promised p95, not average throughput
More replicasImprovement under loadImprovement under loadThe bottleneck is a shared dependency
Trade-off mapInteractive

Good for

Large improvement

Cost

Neutral

Fails when

The task genuinely needs the context

  1. The task genuinely needs the context
  2. Prefixes differ per user or request
  3. Quality floor is already tight
  4. Draft model acceptance rate is low
  5. You promised p95, not average throughput

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Shorter prompt is good for Large improvement, costs Neutral, and fails when The task genuinely needs the context. Prefix / prompt caching is good for Large on repeated prefixes, costs Neutral, and fails when Prefixes differ per user or request. Smaller or distilled model is good for Improvement, costs Improvement, and fails when Quality floor is already tight. Speculative decoding is good for Neutral to slightly worse, costs Large improvement, and fails when Draft model acceptance rate is low. Larger batches, higher utilisation is good for Worse tails, costs Worse tails, and fails when You promised p95, not average throughput

Failure modes

  • Averaging a bimodal distribution. Cache hits and misses produce two clusters, and the mean describes neither. Debug by plotting the TTFT histogram split by cache-hit and queue-depth; if the distribution has two humps, report per-cluster percentiles.
  • Measuring only server-side. Tokens can be buffered, compressed or delayed in transit. Debug by logging the first-byte timestamp at the client and at the edge for the same trace id.
  • Token-count mismatch during decode. Some APIs bill reasoning or cached tokens that never appear as streamed chunks, so "chunks per second" understates real TPOT. Debug by comparing the reported completion token count with observed chunks.
  • Cold start mistaken for steady state. A new replica pays model load and cache warm-up on its first requests. Debug by tagging latency with replica age and excluding the first minutes from steady-state SLOs.

In production

Set separate objectives per route: a support copilot and a batch summariser do not deserve the same TTFT target. Alert on p95 and p99, never on the mean, and page on a burn rate over hours rather than a single spike. See AI SLOs.

Budget the decomposition explicitly. If the product target is 2.5 s to the first token, allocate retrieval 200 ms, reranking 120 ms, prefill and queue 1.2 s, and leave the remainder as headroom. When one stage exceeds its allocation, you know where to look before opening a profiler.

Cap output length at the product layer, not only at the API. max_output_tokens protects spend; a UI that stops rendering at 400 tokens protects the user's patience.

Common misconceptions

People think: Streaming makes the model faster.

Actually: Streaming changes when the first token is visible, not how long generation takes. Total time is unchanged; perceived responsiveness improves because the user sees progress.

People think: Low average latency means a good experience.

Actually: Users remember the slow requests. A 300 ms mean with a 6 s p99 is a product with a latency problem, and the mean will never show it.

Interview angle

Expect "the chatbot feels slow, what do you do?" A strong answer separates TTFT from ITL, asks for the histogram rather than the mean, and checks the boundaries first (proxy buffering, region, cold replicas) before touching the model. Then it prices the fixes: shorter prompts and caching for TTFT, a smaller or speculative model for ITL, more replicas for queueing. Finish with the trade-off: batching raises throughput and worsens tails. See queueing and throughput and the production interview category.

Sources