Course contentsShow
← AI Engineering
Lesson 1063 of 1,88626. Self-Hosted LLM DeploymentFree lesson

GPU Memory Hierarchy and Bandwidth

Understand how VRAM capacity, bandwidth, and memory architecture affect model loading and inference speed.

GPU Memory Hierarchy and Bandwidth

You'll learn how different GPU memory layers and their bandwidth determine how fast your LLM actually runs, even if it fits in VRAM.

Why Memory Hierarchy Matters

Getting your model loaded into VRAM is only half the battle. GPUs organize memory in multiple layers—each with different speeds—and how data moves through these layers dramatically affects inference performance.

Think of it like a kitchen: your ingredients might all fit in the house (VRAM capacity), but performance depends on whether they're in the pantry, the counter, or already in your hand. The closer and faster the memory, the better.

GPU Memory Layers

Modern GPUs have several memory tiers:

  1. Global Memory (VRAM) – The largest pool (e.g., 24GB on an A10G). Slowest to access but holds your model weights, activations, and KV cache.

  2. L2 Cache – Shared across the GPU (typically 40-60MB). Acts as a staging area between global memory and compute cores.

  3. L1 Cache / Shared Memory – Fast, small memory per streaming multiprocessor (SM). Critical for attention operations and matrix multiplications.

  4. Registers – Fastest but tiny storage directly in compute units.

Bandwidth: The Real Bottleneck

Memory bandwidth (measured in GB/s) determines how quickly data moves between these layers. An A100 has ~1.5TB/s bandwidth; a consumer RTX 4090 has ~1TB/s.

For LLMs, bandwidth often matters more than raw compute power because you're constantly loading weights. This is why batch size and KV cache optimization are crucial—they reduce memory trips.

Key Takeaway: GPU inference speed depends not just on VRAM capacity, but on memory bandwidth and how efficiently your serving framework moves data through the GPU's memory hierarchy.