seniorReportedcurated
What is FlashAttention, why is it exact rather than approximate, and what does it not fix?
30-second answer
- Standard attention writes the n×n score matrix to HBM and reads it back for the softmax and the value product, so at long context it is limited by memory traffic rather than FLOPs.
- FlashAttention tiles Q, K and V into blocks sized to on-chip memory, keeps an online softmax with a running maximum and normaliser, and rescales partial output when a new maximum arrives.
- It is exact: the arithmetic is the same attention with a different memory schedule, which is why it is typically 2–3× faster with far lower memory rather than an approximation.
- It still performs O(n²) arithmetic, so it raises the feasible context length and lowers the constant without changing the asymptotics.
- The costs are kernel complexity, restricted head dimensions and dtypes, and a silent fallback to the slow path when the configuration is unsupported.
- It also does not shrink the KV cache, so the speedup concentrates in prefill and training while decode stays bound by cache reads.
Likely follow-ups
Open a question to rehearse the answer.
Why is online softmax necessary for tiling?
- A tiled kernel sees key blocks one at a time, so it cannot know the global maximum needed for stable exponentiation until every block has been read.
- The online softmax maintains a running maximum and running sum, and rescales the accumulated output whenever a larger maximum arrives.
- Without that rescaling the partial sums would be computed against different normalisers and the result would not match a full softmax.
- Recomputing intermediate tiles in the backward pass is the other half of the trade: it avoids storing the score matrix by doing the arithmetic again.
Does FlashAttention reduce the KV cache?
- No: it changes how attention is computed, not what is stored, so keys and values for every cached token are still held per layer.
- The memory it removes is the transient n×n score matrix, which matters in prefill and training rather than in decode.
- Cache size still follows 2 × layers × KV heads × head dimension × tokens × bytes, so GQA, quantisation or paging are the levers that move it.
- Expecting inference memory to drop is the common error, because at long context the cache and not the scores is the binding constraint.
Red flags
- Calls it sparse or approximate attention
- Claims it makes attention linear
Concepts this tests
Tags
Source: www.designgurus.io