staffReportedcurated
How can you increase the context length of an LLM, and what actually limits it?
30-second answer
- Long context is bounded by O(n²) attention compute, an O(n) KV cache that grows with context and batch, and positional generalisation past the trained length; FlashAttention removes score-matrix memory and lowers the constant, not the quadratic term.
- Levers on bytes per token are GQA and KV quantisation; paging changes how memory is allocated rather than how much each token costs; fused kernels cut memory traffic and windowed attention cuts the pairs computed.
- Levers on positional generalisation are RoPE base or frequency scaling with a brief fine-tune, while retrieval or compaction sidesteps the long window instead of extending it.
- None of that changes the provider's nominal rate, but a longer window still costs more per request because there are more tokens to process and to cache.
- Validate with needle-in-a-haystack and task evals rather than perplexity, because naive extension can leave loss healthy while long-range recall collapses.
Likely follow-ups
Open a question to rehearse the answer.
Why does a longer window raise cost even when quality holds?
- Attention compute grows quadratically with sequence length, so doubled context costs roughly four times the prefill work.
- The KV cache grows linearly with tokens and multiplies by concurrent requests, which caps how many sequences fit on a device.
- Decode reads the whole cache on every step, so a longer context raises inter-token latency as well as memory.
- A bigger window also delays time-to-first-token, because prefill must process every prompt token before any output exists.
Why might a 1M-token window still perform worse than retrieval?
- Models attend unevenly across long contexts and often degrade for evidence in the middle, so a fact can be present and still missed.
- Retrieval filters by permission and recency before generation, while a long prompt carries everything and can arrive with conflicting versions.
- Cost and latency scale with the tokens you send, and beyond a point extra passages act as distractors rather than evidence.
- The ceiling is the same either way: if the right passage is not in the context, no prompt fixes the answer.
Red flags
- Says context length is just a config value you raise
- Validates extension only with training loss or perplexity
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?