Flash Attention 2 Improvements
You'll learn: How Flash Attention 2 refines the original algorithm through better parallelism and fewer wasted operations to deliver even faster attention computation.
Beyond the Original Flash Attention
While the original Flash Attention dramatically reduced memory I/O by computing attention in tiled blocks (as you learned previously), Flash Attention 2 optimizes how those computations execute on modern GPUs. Think of it as going from a well-organized assembly line to one where workers coordinate more efficiently and eliminate redundant steps.
Key Algorithmic Improvements
Reduced Non-Matmul FLOPs
Flash Attention 2 minimizes operations outside the core matrix multiplications. The original algorithm performed some redundant rescaling operations when updating running statistics for softmax. FA2 restructures the computation to avoid recalculating these values, cutting non-matmul FLOPs by approximately 2×.
Enhanced Parallelism
The new implementation parallelizes work across the sequence length dimension rather than the batch and head dimensions. This change better utilizes GPU thread blocks, especially for longer sequences where there's more work to distribute. Each thread block now processes a larger, contiguous chunk of the attention matrix.
Work Partitioning
FA2 adjusts how work divides among GPU warps. Instead of splitting queries and keys/values equally, it assigns different amounts of work based on the actual computational cost, reducing idle time and improving occupancy on the streaming multiprocessors.
Forward-Backward Consistency
The backward pass now uses the same tiling strategy as the forward pass, improving code reuse and cache behavior during gradient computation—critical for training large models efficiently.
Key Takeaway: Flash Attention 2 achieves 2-3× speedup over the original through smarter parallelization across sequence length, reduced redundant operations, and better GPU resource utilization—all while maintaining the same memory-efficient tiling approach.