Skip to content
AI Engineer Guide.
staffReportedcurated

How can you increase the context length of an LLM, and what actually limits it?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why does a longer window raise cost even when quality holds?
  • Attention compute grows quadratically with sequence length, so doubled context costs roughly four times the prefill work.
  • The KV cache grows linearly with tokens and multiplies by concurrent requests, which caps how many sequences fit on a device.
  • Decode reads the whole cache on every step, so a longer context raises inter-token latency as well as memory.
  • A bigger window also delays time-to-first-token, because prefill must process every prompt token before any output exists.
Why might a 1M-token window still perform worse than retrieval?
  • Models attend unevenly across long contexts and often degrade for evidence in the middle, so a fact can be present and still missed.
  • Retrieval filters by permission and recency before generation, while a long prompt carries everything and can arrive with conflicting versions.
  • Cost and latency scale with the tokens you send, and beyond a point extra passages act as distractors rather than evidence.
  • The ceiling is the same either way: if the right passage is not in the context, no prompt fixes the answer.

Red flags

  • Says context length is just a config value you raise
  • Validates extension only with training loss or perplexity

Concepts this tests

Tags

Source: github.com

More in LLM & transformer fundamentals