Common GPU Memory Pitfalls
You'll learn how to identify and fix the three most common causes of GPU out-of-memory errors in PyTorch.
Now that you understand GPU memory fundamentals and monitoring, let's tackle the mistakes that actually crash your training runs.
The Three Main Culprits
1. Retained Computation Graphs
When you call .backward(), PyTorch builds a graph to compute gradients. If you accidentally keep references to intermediate tensors or loss values without detaching them, these graphs stay in memory forever.
The fix: Use .item() to extract scalar values for logging, or wrap evaluation code in torch.no_grad().
2. Batch Size Too Large
Think of GPU memory like a parking lot—if you try to park too many cars (samples) at once, you run out of spaces. Large batch sizes mean more activations stored during the forward pass.
The fix: Reduce batch size, or use gradient accumulation to simulate larger batches while processing smaller chunks.
3. Inefficient CPU-GPU Transfers
Moving data between CPU and GPU is like shipping packages—every transfer has overhead. Creating tensors on CPU then moving them to GPU repeatedly wastes time and can cause memory fragmentation.
The fix: Create tensors directly on GPU when possible, use pin_memory=True in DataLoaders, and minimize the number of transfers per batch.
Real-World Example Pattern
The classic mistake: saving entire loss tensors in a list during training instead of just their values. Each tensor keeps its computation graph alive, eating memory until you crash.
Key Takeaway: Most GPU memory issues stem from unintentionally holding onto computation graphs, using batches larger than GPU memory allows, or inefficient data transfer patterns—all preventable with proper tensor management and device-aware coding.