Course contentsShow
← Machine Learning and Deep Learning
Lesson 820 of 3,53819. PyTorch FundamentalsFree lesson

pin_memory and GPU Transfer Optimization

Optimize host-to-device transfers using pinned memory for faster batch movement to GPU.

pin_memory and GPU Transfer Optimization

What You'll Learn

You'll learn how to use pinned memory to accelerate data transfer from CPU to GPU, reducing the bottleneck in your training pipeline.

The Core Idea

When your DataLoader prepares batches on the CPU and your model trains on the GPU, PyTorch must copy data from system RAM to GPU memory. Normally, this copying happens from pageable memory, which the operating system can move around, making transfers slower.

Pinned memory (also called page-locked memory) is a special region of RAM that stays in a fixed location. The GPU can access it directly via DMA (Direct Memory Access), bypassing the CPU entirely during transfer. This dramatically speeds up host-to-device copies.

Think of it like the difference between moving furniture from a storage unit (pageable memory) versus from your driveway (pinned memory) — the driveway is already accessible to the moving truck, so loading is instant.

How to Use It

Simply add pin_memory=True when creating your DataLoader:

from torch.utils.data import DataLoader

train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=64,
    shuffle=True,
    num_workers=4,
    pin_memory=True  # Enable pinned memory
)

When enabled, batches are allocated in pinned memory, and calling .to(device) on your GPU becomes much faster.

When to Use It

  • Always use it when training on GPU
  • Don't use it for CPU-only training (adds overhead without benefit)
  • Pair with num_workers > 0 for maximum pipeline efficiency

Trade-off

Pinned memory is a limited resource. Allocating too much can reduce system performance, but typical batch sizes are fine.

Key Takeaway: Set pin_memory=True in your DataLoader when training on GPU to accelerate data transfer by using page-locked memory that the GPU can access directly.