GPU Memory Management Fundamentals
What you'll learn: How PyTorch allocates and manages GPU memory, including why freeing a tensor doesn't immediately return memory to the system.
The GPU Memory Problem
When you move tensors and models to the GPU, they consume VRAM (video memory). Unlike CPU RAM, GPU memory is much more limited—typically 8-24 GB on consumer cards. Understanding how PyTorch manages this scarce resource is critical to avoiding "out of memory" errors.
The Caching Allocator
PyTorch uses a caching allocator to manage GPU memory efficiently. Think of it like renting storage units: when you're done with one unit, you don't immediately return the key to the landlord. Instead, you keep it in case you need storage again soon.
Here's what happens:
- First allocation: PyTorch requests a block of GPU memory from CUDA
- Tensor deletion: When you delete a tensor or it goes out of scope, PyTorch marks that memory as "free" but doesn't return it to the GPU
- Reuse: The next tensor allocation tries to reuse cached memory before requesting new blocks
- Why? Requesting memory from CUDA is slow. Caching dramatically speeds up training loops where tensors are created and destroyed repeatedly
The Implications
This means torch.cuda.memory_allocated() shows memory actually used by tensors, while torch.cuda.memory_reserved() shows memory held in PyTorch's cache. Even after deleting all tensors, reserved memory stays high until you explicitly call torch.cuda.empty_cache().
Key Takeaway: PyTorch caches GPU memory after tensors are freed to avoid expensive reallocation, so memory isn't immediately returned to the system—use empty_cache() only when necessary, as normal caching improves performance.