Course contentsShow
← Machine Learning and Deep Learning
Lesson 1680 of 3,53836. LLM Inference OptimizationFree lesson

IO-Awareness and GPU Memory Hierarchy

How GPU memory hierarchy (HBM vs SRAM) affects attention performance and the importance of minimizing memory transfers.

IO-Awareness and GPU Memory Hierarchy

What you'll learn: How the structure of GPU memory directly determines whether your attention computation is fast or painfully slow.

The Memory Speed Gap

Your GPU has two main types of memory with vastly different speeds:

  • HBM (High Bandwidth Memory): Large but slow. This is where your model weights and KV cache live—think of it as a distant warehouse.
  • SRAM (on-chip cache): Tiny but blazingly fast. This is where actual computation happens—like a chef's cutting board.

The problem? Moving data between HBM and SRAM is expensive—often more expensive than the actual math operations!

Why This Matters for Attention

Standard attention implementations are compute-bound on paper but memory-bound in reality. Here's what happens:

  1. Load Query matrix from HBM → SRAM
  2. Load Key matrix from HBM → SRAM
  3. Compute attention scores (fast!)
  4. Write scores back to HBM (slow!)
  5. Load scores again from HBM → SRAM
  6. Load Value matrix from HBM → SRAM
  7. Compute final output

Notice the pattern? You're constantly shuttling data between slow and fast memory. Each round trip to HBM is like driving across town instead of grabbing ingredients from your kitchen counter.

IO-Aware Optimization

IO-aware algorithms minimize these transfers by:

  • Fusing operations to keep intermediate results in SRAM
  • Computing outputs in blocks that fit in fast memory
  • Never writing unnecessary intermediate results to HBM

This is the core insight behind Flash Attention: reorganize the computation order to reduce memory trips, even if it means doing slightly more math.

Key Takeaway: On modern GPUs, where your data lives matters more than how many operations you perform—optimizing memory transfers, not FLOPs, is the path to faster inference.