Why it matters
A naive attention implementation writes an N by N score matrix to off-chip memory and reads it back for softmax and the value product. At long context that matrix dominates both memory and time, even though the arithmetic is modest. FlashAttention reorganises the loop so partial results stay on chip, which makes long-context attention feasible on the same hardware without changing what attention computes.
Mental model
It is cooking in batches that fit the counter instead of laying every ingredient out on the table first. The analogy breaks because the recipe needs a running total, so the cook must keep a corrected sum as each batch arrives rather than simply appending results.
How it works
- 1Tile queries, keys and values into blocks sized to the on-chip memory of the target device.
- 2Compute partial attention for each key tile and maintain a running maximum and normaliser for the softmax.
- 3Rescale the accumulated output when a new maximum arrives, so the final result matches a full softmax.
- 4Apply the causal mask inside the tile loop so masked blocks are skipped rather than computed and discarded.
- 5Recompute intermediate tiles in the backward pass instead of storing the score matrix, trading arithmetic for memory.
- 6Confirm at runtime that the fused kernel is selected for the head dimension and dtype in use, rather than a fallback.
Step 1 of 6
Tile queries, keys and values into blocks sized to the on-chip memory of the target device.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Tile queries, keys and values into blocks sized to the on-chip memory of the target devic… 2. Compute partial attention for each key tile and maintain a running maximum and normaliser… 3. Rescale the accumulated output when a new maximum arrives, so the final result matches a … 4. Apply the causal mask inside the tile loop so masked blocks are skipped rather than compu… 5. Recompute intermediate tiles in the backward pass instead of storing the score matrix, tr… 6. Confirm at runtime that the fused kernel is selected for the head dimension and dtype in …
- Tile queries, keys and values into blocks sized to the on-chip memory of the target device.
- Compute partial attention for each key tile and maintain a running maximum and normaliser for the softmax.
- Rescale the accumulated output when a new maximum arrives, so the final result matches a full softmax.
- Apply the causal mask inside the tile loop so masked blocks are skipped rather than computed and discarded.
- Recompute intermediate tiles in the backward pass instead of storing the score matrix, trading arithmetic for memory.
- Confirm at runtime that the fused kernel is selected for the head dimension and dtype in use, rather than a fallback.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Fused kernel | Lower memory, faster long-context attention | Restricted head dimensions and dtypes | The model uses an unsupported configuration |
| Recompute in backward | No stored score matrix | Extra arithmetic during training | Training is compute-bound already |
| Long context enabled | Fewer truncations | Attention cost still grows with sequence length | Context is far longer than the task needs |
| Fallback implementation | Works for any shape | Materialises the score matrix | Context length is the reason for choosing the kernel |
Good for
Lower memory, faster long-context attention
Cost
Restricted head dimensions and dtypes
Fails when
The model uses an unsupported configuration
- The model uses an unsupported configuration
- Training is compute-bound already
- Context is far longer than the task needs
- Context length is the reason for choosing the kernel
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Fused kernel is good for Lower memory, faster long-context attention, costs Restricted head dimensions and dtypes, and fails when The model uses an unsupported configuration. Recompute in backward is good for No stored score matrix, costs Extra arithmetic during training, and fails when Training is compute-bound already. Long context enabled is good for Fewer truncations, costs Attention cost still grows with sequence length, and fails when Context is far longer than the task needs. Fallback implementation is good for Works for any shape, costs Materialises the score matrix, and fails when Context length is the reason for choosing the kernel
Failure modes
- Silent fallback. The kernel rejects the head dimension and the framework uses the slow path, so the expected speedup never appears. Debug: profile kernel names and check for an attention fallback in the trace.
- Numerical mismatch in low precision. Outputs diverge from the reference implementation at long context. Debug: compare logits against a float32 reference on the same prompt.
- Memory expectation error. Teams expect inference memory to drop, but decode is dominated by the cache, not the score matrix. Debug: separate weight, cache and activation memory in the profiler.
In production
Watch time-to-first-token against prompt length, since prefill is where the kernel matters most, and record which attention implementation was loaded at startup. Pin the kernel version with the model, because a driver or library change can silently move you to the fallback.
Common misconceptions
People think: FlashAttention approximates attention to make it cheaper.
Actually: It computes the same result with a different memory schedule; approximations are a separate family of methods.
People think: It speeds up every phase of serving.
Actually: The gain is largest in prefill and training; decode is bandwidth-bound on the cache and benefits far less.
Interview angle
Explain the input-output bottleneck, the tiling strategy and the online softmax, then say why the result is exact. A strong answer names the configuration that silently disables the kernel. Weak answers call it a faster attention formula.
Sources
- Dao et al., FlashAttention — input-output-aware tiling and the online softmax.
- Dao, FlashAttention-2 — improved work partitioning and parallelism.
- Dao-AILab, flash-attention — supported head dimensions and dtypes.