Course contentsShow
← Machine Learning and Deep Learning
Lesson 848 of 3,53819. PyTorch FundamentalsFree lesson

Common GPU Memory Pitfalls

Identify and fix memory leaks from retained computation graphs, large batch sizes, and CPU-GPU data transfers.

Common GPU Memory Pitfalls

You'll learn how to identify and fix the three most common causes of GPU out-of-memory errors in PyTorch.

Now that you understand GPU memory fundamentals and monitoring, let's tackle the mistakes that actually crash your training runs.

The Three Main Culprits

1. Retained Computation Graphs

When you call .backward(), PyTorch builds a graph to compute gradients. If you accidentally keep references to intermediate tensors or loss values without detaching them, these graphs stay in memory forever.

The fix: Use .item() to extract scalar values for logging, or wrap evaluation code in torch.no_grad().

2. Batch Size Too Large

Think of GPU memory like a parking lot—if you try to park too many cars (samples) at once, you run out of spaces. Large batch sizes mean more activations stored during the forward pass.

The fix: Reduce batch size, or use gradient accumulation to simulate larger batches while processing smaller chunks.

3. Inefficient CPU-GPU Transfers

Moving data between CPU and GPU is like shipping packages—every transfer has overhead. Creating tensors on CPU then moving them to GPU repeatedly wastes time and can cause memory fragmentation.

The fix: Create tensors directly on GPU when possible, use pin_memory=True in DataLoaders, and minimize the number of transfers per batch.

Real-World Example Pattern

The classic mistake: saving entire loss tensors in a list during training instead of just their values. Each tensor keeps its computation graph alive, eating memory until you crash.

Key Takeaway: Most GPU memory issues stem from unintentionally holding onto computation graphs, using batches larger than GPU memory allows, or inefficient data transfer patterns—all preventable with proper tensor management and device-aware coding.