Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 32 of 33

Long-context attention

0 of 33 complete

Why it matters

Doubling context quadruples attention work and grows the cache, so the naive path becomes uneconomical before it becomes impossible. Teams then face two separate problems: making the arithmetic affordable, and making the model actually use positions it never saw in training. Solutions that fix one and not the other produce a model that accepts long inputs and ignores their middle.

Mental model

Long-context attention is reading a long report through a movable window and a set of sticky notes: you see detail locally and rely on summaries for the rest. The metaphor breaks because the model has no explicit summary — sparse and windowed patterns are fixed or learned, and nothing guarantees the distant fact you need was ever read.

How it works

Sparse pattern · window of one, plus a global tokenInteractive
00.600outside pattern
query ↓key →
t1t2t3t4t5Σ
t11.000
t21.000
t31.000
t41.000
t51.000

Hover or focus a cell to read its exact value and printed row total. Dashed cells are removed by the sparse pattern before normalisation.

window
1 neighbour each side
global tokens
1 · t1
cells kept
25 − 6 = 19
cells per row
3 to 5

Step 1 of 4

Pattern on · row t4

t4 keeps t3, itself, t5 and the global t1: 0.233, 0.349, 0.279 and 0.139, which renormalises to 1.000. The one other key is not scored at all.

Next: The global token

Step 1/4 · Pattern on · row t4

Figure. Five queries over five keys with a sliding window of one neighbour each side plus a global first token, illustrative weights. Dashed cells are outside the pattern and are never scored; each row renormalises over what it keeps, so every printed row still totals 1.000. The pattern is the claim: the work per row is bounded by the pattern, not by the length.

What to notice. t5 can read t4, itself and the global t1, and the far key t3 is outside its pattern whatever it contains. Compare with the mask off: every row still totals 1.000, but t5 now puts weight on t3. A sparse pattern is therefore a recall decision as well as a cost decision, and the global token is the only row and column that reaches the whole sequence, which is why a far key can still cross from t1 into a later row. The figure shows one pattern at one length; real variants differ in window size, number of global tokens and whether the pattern is learned.

Text version

A five by five attention matrix. Rows are queries t1 to t5 and columns the same five keys. The sliding-window-plus-global pattern is on by default: 6 cells are dashed and empty, and each row keeps the first column, the diagonal and one neighbour each side, while row t1 keeps every column. Row t1 prints 0.300, 0.240, 0.180, 0.160 and 0.120; t2 prints 0.243, 0.433 and 0.324; t3 prints 0.167, 0.238, 0.357 and 0.238; t4 prints 0.139 at t1, 0.233 at t3, 0.349 at t4 and 0.279 at t5; t5 prints 0.139 at t1, 0.361 at t4 and 0.500 at t5. Every row-total cell reads 1.000. Steps highlight the t4 row, the global column, the t5 row and the mask off state, where every cell carries weight and rows still total 1.000.

  1. Measure the actual length distribution of your inputs before choosing a technique; most products need a smaller window than they assume.
  2. Reduce cost first with fused kernels and cache management, which preserve exact attention within the window.
  3. If length still dominates, choose a pattern: sliding window for local structure, sparse global tokens for anchors, low-rank projection for dense summarisation.
  4. Adapt position encoding to the target length, using frequency-base adjustment or interpolation, and train on long examples.
  5. Evaluate with position-controlled probes that place the answer at the start, middle and end of the context.
  6. Compare against chunked retrieval, which is often cheaper and more accurate when only a few passages matter.

Trade-offs

ChoiceGood forCostFails when
Sliding windowCheap, local coherence, streamingCannot reach distant evidence in one layerThe answer depends on a far-away fact
Sparse global plus localSome long-range reach at lower costPattern design and kernel complexityRelevant positions fall outside the fixed pattern
Low-rank or linear attentionSub-quadratic scalingQuality loss on exact recallTasks need precise copying or lookup
Retrieval over chunksCheap, auditable, scales with corpusMisses cross-passage reasoningThe question requires synthesising the whole document
Trade-off mapInteractive

Good for

Cheap, local coherence, streaming

Cost

Cannot reach distant evidence in one layer

Fails when

The answer depends on a far-away fact

  1. The answer depends on a far-away fact
  2. Relevant positions fall outside the fixed pattern
  3. Tasks need precise copying or lookup
  4. The question requires synthesising the whole document

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Sliding window is good for Cheap, local coherence, streaming, costs Cannot reach distant evidence in one layer, and fails when The answer depends on a far-away fact. Sparse global plus local is good for Some long-range reach at lower cost, costs Pattern design and kernel complexity, and fails when Relevant positions fall outside the fixed pattern. Low-rank or linear attention is good for Sub-quadratic scaling, costs Quality loss on exact recall, and fails when Tasks need precise copying or lookup. Retrieval over chunks is good for Cheap, auditable, scales with corpus, costs Misses cross-passage reasoning, and fails when The question requires synthesising the whole document

Failure modes

  • Advertised length exceeds trained length. The model accepts long inputs but quality falls off a cliff past the training range. Debug: evaluate loss and task accuracy in length buckets and find the knee.
  • Middle-of-context neglect. Facts placed centrally are ignored while the same facts at the edges are used. Debug: run a position-controlled probe that moves one supporting sentence across the context.
  • Approximation without recall check. A sparse or low-rank variant passes average evals and misses exact matches. Debug: measure exact-match retrieval of planted strings against a full-attention baseline.

In production

Track quality and latency by context-length bucket, and cap the window at the length you have actually evaluated. For long-document products, keep a retrieval path as a fallback and compare its answers against the long-context path on the same questions.

Common misconceptions

People think: A larger context window replaces retrieval.

Actually: A window raises the ceiling on what can be read but not the reliability of finding one fact among many, and it costs more per request than selecting passages.

People think: Long-context support is purely a kernel problem.

Actually: Kernels make the arithmetic affordable; position encoding and training data determine whether the model can use the extra positions at all.

Interview angle

Treat the compute problem and the generalisation problem as separate, name concrete approximation families, and describe position-controlled evaluation. Compare long context with retrieval on cost and accuracy. Weak answers quote a window size without saying what was trained, what it cost, or how middle-context behaviour was measured.

Sources