Course contentsShow
← Machine Learning and Deep Learning
Lesson 847 of 3,53819. PyTorch FundamentalsFree lesson

Monitoring GPU Memory Usage

Use torch.cuda.memory_allocated(), memory_reserved(), and nvidia-smi to track GPU memory consumption.

Monitoring GPU Memory Usage

What you'll learn: How to track and inspect how much GPU memory your PyTorch models and tensors are actually consuming.

Why Monitor GPU Memory?

When training deep learning models on GPUs, you're working with a finite memory budget. Think of GPU memory like RAM in your computer—except it's much more limited and more expensive. Running out of GPU memory causes crashes with cryptic "CUDA out of memory" errors. Monitoring helps you understand what's using memory and avoid these issues.

Three Essential Tools

1. torch.cuda.memory_allocated()

Returns the current memory occupied by tensors in bytes. This is the actual data you've created:

import torch

# Check memory before creating tensors
print(f"Before: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

x = torch.randn(10000, 10000, device='cuda')

# Check memory after
print(f"After: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

2. torch.cuda.memory_reserved()

Shows memory reserved by PyTorch's caching allocator. PyTorch reserves more memory than immediately needed for efficiency—like reserving a large table at a restaurant even if some seats are empty:

print(f"Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")

3. nvidia-smi Command

Run this in your terminal (not Python) to see GPU utilization and memory across all processes:

nvidia-smi

This shows the "big picture" of what's happening on your GPU hardware.

Practical Usage

Check memory at key points: before training, after model creation, during the forward pass, and after backward passes. This helps identify memory leaks or unexpectedly large allocations.

Key Takeaway: Use torch.cuda.memory_allocated() for tensor memory, memory_reserved() for PyTorch's cache, and nvidia-smi for system-level GPU monitoring to track and debug memory usage effectively.