Course contentsShow
← Machine Learning and Deep Learning
Lesson 1682 of 3,53836. LLM Inference OptimizationFree lesson

Softmax Computation with Tiling

How Flash Attention computes numerically stable softmax in blocks using online normalization tricks.

Softmax Computation with Tiling

What you'll learn: How Flash Attention computes numerically stable softmax in blocks without loading all attention scores into memory at once.

The Challenge

In standard attention, you compute all attention scores, apply softmax across the entire sequence, then multiply by values. But softmax requires knowing the maximum value and sum across all elements—which seems to require loading everything into memory first. Flash Attention tiles the computation into blocks, so how does it calculate a global softmax from partial blocks?

Online Normalization Trick

Flash Attention uses an online softmax algorithm that incrementally updates the softmax as it processes each tile:

  1. Track running statistics: As you process each block of attention scores, maintain the current maximum and current sum of exponentials
  2. Rescale previous results: When a new block has a larger maximum, rescale all previously computed softmax outputs using the difference in max values
  3. Accumulate incrementally: Add the new block's contribution to the running sum

Think of it like computing a class average as students trickle in: you don't wait for everyone—you adjust your running average with each new score.

Why This Matters

Without this technique, you'd need to:

  • Load all attention scores into HBM (slow, memory-intensive)
  • Compute softmax in one pass
  • Then load values

With tiling and online normalization:

  • Process attention scores and values block-by-block in SRAM (fast)
  • Never materialize the full attention matrix
  • Maintain numerical stability through careful max-tracking

The math ensures that the final result is exactly the same as standard softmax, but the computation path dramatically reduces memory reads/writes.

Key Takeaway: Flash Attention's tiled softmax uses online normalization to compute numerically stable softmax incrementally across blocks, avoiding the need to load all attention scores into memory simultaneously—this is the core trick enabling IO-efficient attention.