Course contentsShow
← Machine Learning and Deep Learning
Lesson 1683 of 3,53836. LLM Inference OptimizationFree lesson

Flash Attention 2 Improvements

Algorithmic and implementation optimizations in Flash Attention 2 for better parallelism and reduced operations.

Flash Attention 2 Improvements

You'll learn: How Flash Attention 2 refines the original algorithm through better parallelism and fewer wasted operations to deliver even faster attention computation.

Beyond the Original Flash Attention

While the original Flash Attention dramatically reduced memory I/O by computing attention in tiled blocks (as you learned previously), Flash Attention 2 optimizes how those computations execute on modern GPUs. Think of it as going from a well-organized assembly line to one where workers coordinate more efficiently and eliminate redundant steps.

Key Algorithmic Improvements

Reduced Non-Matmul FLOPs

Flash Attention 2 minimizes operations outside the core matrix multiplications. The original algorithm performed some redundant rescaling operations when updating running statistics for softmax. FA2 restructures the computation to avoid recalculating these values, cutting non-matmul FLOPs by approximately 2×.

Enhanced Parallelism

The new implementation parallelizes work across the sequence length dimension rather than the batch and head dimensions. This change better utilizes GPU thread blocks, especially for longer sequences where there's more work to distribute. Each thread block now processes a larger, contiguous chunk of the attention matrix.

Work Partitioning

FA2 adjusts how work divides among GPU warps. Instead of splitting queries and keys/values equally, it assigns different amounts of work based on the actual computational cost, reducing idle time and improving occupancy on the streaming multiprocessors.

Forward-Backward Consistency

The backward pass now uses the same tiling strategy as the forward pass, improving code reuse and cache behavior during gradient computation—critical for training large models efficiently.

Key Takeaway: Flash Attention 2 achieves 2-3× speedup over the original through smarter parallelization across sequence length, reduced redundant operations, and better GPU resource utilization—all while maintaining the same memory-efficient tiling approach.