pin_memory and GPU Transfer Optimization
What You'll Learn
You'll learn how to use pinned memory to accelerate data transfer from CPU to GPU, reducing the bottleneck in your training pipeline.
The Core Idea
When your DataLoader prepares batches on the CPU and your model trains on the GPU, PyTorch must copy data from system RAM to GPU memory. Normally, this copying happens from pageable memory, which the operating system can move around, making transfers slower.
Pinned memory (also called page-locked memory) is a special region of RAM that stays in a fixed location. The GPU can access it directly via DMA (Direct Memory Access), bypassing the CPU entirely during transfer. This dramatically speeds up host-to-device copies.
Think of it like the difference between moving furniture from a storage unit (pageable memory) versus from your driveway (pinned memory) — the driveway is already accessible to the moving truck, so loading is instant.
How to Use It
Simply add pin_memory=True when creating your DataLoader:
from torch.utils.data import DataLoader
train_loader = DataLoader(
dataset=train_dataset,
batch_size=64,
shuffle=True,
num_workers=4,
pin_memory=True # Enable pinned memory
)
When enabled, batches are allocated in pinned memory, and calling .to(device) on your GPU becomes much faster.
When to Use It
- Always use it when training on GPU
- Don't use it for CPU-only training (adds overhead without benefit)
- Pair with
num_workers > 0for maximum pipeline efficiency
Trade-off
Pinned memory is a limited resource. Allocating too much can reduce system performance, but typical batch sizes are fine.
Key Takeaway: Set pin_memory=True in your DataLoader when training on GPU to accelerate data transfer by using page-locked memory that the GPU can access directly.