Understanding GPU Memory Hierarchy for Inference
You'll learn how different GPU memory types—VRAM, L2 cache, and shared memory—affect inference speed and throughput.
The Memory Speed Pyramid
Think of GPU memory like a restaurant kitchen: the countertop (registers and shared memory) is fastest but smallest, the nearby shelf (L2 cache) is medium-speed and medium-sized, and the walk-in freezer (VRAM) is largest but requires walking back and forth. Data movement between these layers directly impacts how fast your model serves predictions.
The Three Key Memory Types
VRAM (Device Memory): This is your GPU's main memory—typically 8GB to 80GB on modern cards. Model weights, activations, and batched inputs live here. It's large but relatively slow (hundreds of GB/s bandwidth). Reading from VRAM repeatedly during inference creates bottlenecks.
L2 Cache: A 40-80MB buffer sitting between compute cores and VRAM. When the same data is accessed multiple times (like attention weights in transformers), L2 cache can serve it 10-20x faster than VRAM. Cache locality is why compact data layouts matter.
Shared Memory: A small (48-164KB per streaming multiprocessor), ultra-fast scratchpad explicitly controlled by kernels. Custom CUDA kernels use shared memory to stage frequently-accessed data, reducing VRAM trips. This is critical for optimized operations like flash attention.
Impact on Inference
High VRAM usage → more data movement → lower throughput. Poor cache locality → repeated VRAM fetches → higher latency. Optimized kernels leveraging shared memory can deliver 2-5x speedups by keeping "hot" data close to compute units.
Key Takeaway: GPU inference performance depends on minimizing expensive VRAM accesses by maximizing L2 cache hits and using shared memory strategically—understanding this hierarchy helps you profile bottlenecks and choose the right optimization techniques.