GPU Memory Hierarchy and Bandwidth
You'll learn how different GPU memory layers and their bandwidth determine how fast your LLM actually runs, even if it fits in VRAM.
Why Memory Hierarchy Matters
Getting your model loaded into VRAM is only half the battle. GPUs organize memory in multiple layers—each with different speeds—and how data moves through these layers dramatically affects inference performance.
Think of it like a kitchen: your ingredients might all fit in the house (VRAM capacity), but performance depends on whether they're in the pantry, the counter, or already in your hand. The closer and faster the memory, the better.
GPU Memory Layers
Modern GPUs have several memory tiers:
-
Global Memory (VRAM) – The largest pool (e.g., 24GB on an A10G). Slowest to access but holds your model weights, activations, and KV cache.
-
L2 Cache – Shared across the GPU (typically 40-60MB). Acts as a staging area between global memory and compute cores.
-
L1 Cache / Shared Memory – Fast, small memory per streaming multiprocessor (SM). Critical for attention operations and matrix multiplications.
-
Registers – Fastest but tiny storage directly in compute units.
Bandwidth: The Real Bottleneck
Memory bandwidth (measured in GB/s) determines how quickly data moves between these layers. An A100 has ~1.5TB/s bandwidth; a consumer RTX 4090 has ~1TB/s.
For LLMs, bandwidth often matters more than raw compute power because you're constantly loading weights. This is why batch size and KV cache optimization are crucial—they reduce memory trips.
Key Takeaway: GPU inference speed depends not just on VRAM capacity, but on memory bandwidth and how efficiently your serving framework moves data through the GPU's memory hierarchy.