Skip to content
AI Engineer Guide.
seniorReportedcurated

What is FlashAttention, why is it exact rather than approximate, and what does it not fix?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why is online softmax necessary for tiling?
  • A tiled kernel sees key blocks one at a time, so it cannot know the global maximum needed for stable exponentiation until every block has been read.
  • The online softmax maintains a running maximum and running sum, and rescales the accumulated output whenever a larger maximum arrives.
  • Without that rescaling the partial sums would be computed against different normalisers and the result would not match a full softmax.
  • Recomputing intermediate tiles in the backward pass is the other half of the trade: it avoids storing the score matrix by doing the arithmetic again.
Does FlashAttention reduce the KV cache?
  • No: it changes how attention is computed, not what is stored, so keys and values for every cached token are still held per layer.
  • The memory it removes is the transient n×n score matrix, which matters in prefill and training rather than in decode.
  • Cache size still follows 2 × layers × KV heads × head dimension × tokens × bytes, so GQA, quantisation or paging are the levers that move it.
  • Expecting inference memory to drop is the common error, because at long context the cache and not the scores is the binding constraint.

Red flags

  • Calls it sparse or approximate attention
  • Claims it makes attention linear

Concepts this tests

Tags

Source: www.designgurus.io

More in Maths & CS fundamentals