Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 10 of 38

FlashAttention

0 of 38 complete

Why it matters

A naive attention implementation writes an N by N score matrix to off-chip memory and reads it back for softmax and the value product. At long context that matrix dominates both memory and time, even though the arithmetic is modest. FlashAttention reorganises the loop so partial results stay on chip, which makes long-context attention feasible on the same hardware without changing what attention computes.

Mental model

It is cooking in batches that fit the counter instead of laying every ingredient out on the table first. The analogy breaks because the recipe needs a running total, so the cook must keep a corrected sum as each batch arrives rather than simply appending results.

How it works

Mechanism, step by stepInteractive
  1. 1Tile queries, keys and values into blocks sized to the on-chip memory of the target device.
  2. 2Compute partial attention for each key tile and maintain a running maximum and normaliser for the softmax.
  3. 3Rescale the accumulated output when a new maximum arrives, so the final result matches a full softmax.
  4. 4Apply the causal mask inside the tile loop so masked blocks are skipped rather than computed and discarded.
  5. 5Recompute intermediate tiles in the backward pass instead of storing the score matrix, trading arithmetic for memory.
  6. 6Confirm at runtime that the fused kernel is selected for the head dimension and dtype in use, rather than a fallback.

Step 1 of 6

Tile queries, keys and values into blocks sized to the on-chip memory of the target device.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Tile queries, keys and values into blocks sized to the on-chip memory of the target devic… 2. Compute partial attention for each key tile and maintain a running maximum and normaliser… 3. Rescale the accumulated output when a new maximum arrives, so the final result matches a … 4. Apply the causal mask inside the tile loop so masked blocks are skipped rather than compu… 5. Recompute intermediate tiles in the backward pass instead of storing the score matrix, tr… 6. Confirm at runtime that the fused kernel is selected for the head dimension and dtype in …

  1. Tile queries, keys and values into blocks sized to the on-chip memory of the target device.
  2. Compute partial attention for each key tile and maintain a running maximum and normaliser for the softmax.
  3. Rescale the accumulated output when a new maximum arrives, so the final result matches a full softmax.
  4. Apply the causal mask inside the tile loop so masked blocks are skipped rather than computed and discarded.
  5. Recompute intermediate tiles in the backward pass instead of storing the score matrix, trading arithmetic for memory.
  6. Confirm at runtime that the fused kernel is selected for the head dimension and dtype in use, rather than a fallback.

Trade-offs

ChoiceGood forCostFails when
Fused kernelLower memory, faster long-context attentionRestricted head dimensions and dtypesThe model uses an unsupported configuration
Recompute in backwardNo stored score matrixExtra arithmetic during trainingTraining is compute-bound already
Long context enabledFewer truncationsAttention cost still grows with sequence lengthContext is far longer than the task needs
Fallback implementationWorks for any shapeMaterialises the score matrixContext length is the reason for choosing the kernel
Trade-off mapInteractive

Good for

Lower memory, faster long-context attention

Cost

Restricted head dimensions and dtypes

Fails when

The model uses an unsupported configuration

  1. The model uses an unsupported configuration
  2. Training is compute-bound already
  3. Context is far longer than the task needs
  4. Context length is the reason for choosing the kernel

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Fused kernel is good for Lower memory, faster long-context attention, costs Restricted head dimensions and dtypes, and fails when The model uses an unsupported configuration. Recompute in backward is good for No stored score matrix, costs Extra arithmetic during training, and fails when Training is compute-bound already. Long context enabled is good for Fewer truncations, costs Attention cost still grows with sequence length, and fails when Context is far longer than the task needs. Fallback implementation is good for Works for any shape, costs Materialises the score matrix, and fails when Context length is the reason for choosing the kernel

Failure modes

  • Silent fallback. The kernel rejects the head dimension and the framework uses the slow path, so the expected speedup never appears. Debug: profile kernel names and check for an attention fallback in the trace.
  • Numerical mismatch in low precision. Outputs diverge from the reference implementation at long context. Debug: compare logits against a float32 reference on the same prompt.
  • Memory expectation error. Teams expect inference memory to drop, but decode is dominated by the cache, not the score matrix. Debug: separate weight, cache and activation memory in the profiler.

In production

Watch time-to-first-token against prompt length, since prefill is where the kernel matters most, and record which attention implementation was loaded at startup. Pin the kernel version with the model, because a driver or library change can silently move you to the fallback.

Common misconceptions

People think: FlashAttention approximates attention to make it cheaper.

Actually: It computes the same result with a different memory schedule; approximations are a separate family of methods.

People think: It speeds up every phase of serving.

Actually: The gain is largest in prefill and training; decode is bandwidth-bound on the cache and benefits far less.

Interview angle

Explain the input-output bottleneck, the tiling strategy and the online softmax, then say why the result is exact. A strong answer names the configuration that silently disables the kernel. Weak answers call it a faster attention formula.

Sources