Course contentsShow
← Machine Learning and Deep Learning
Lesson 850 of 3,53819. PyTorch FundamentalsFree lesson

Optimizing CPU-GPU Data Transfer

Use pinned memory, non-blocking transfers, and efficient data loading to minimize CPU-GPU bottlenecks.

Optimizing CPU-GPU Data Transfer

What you'll learn: How to minimize the bottleneck caused by moving data between CPU and GPU using pinned memory, non-blocking transfers, and efficient data loading strategies.

Why Data Transfer is a Bottleneck

Even though GPUs compute incredibly fast, they can sit idle waiting for data to arrive from the CPU. The data transfer between CPU RAM and GPU memory happens over the PCIe bus, which is much slower than either device's internal memory. It's like having a race car (GPU) stuck in traffic (data transfer) — the raw speed doesn't matter if you can't get fuel to the engine.

Three Key Optimization Techniques

Pinned Memory: By default, CPU data lives in pageable memory, which the operating system can move around. Pinned (or page-locked) memory stays in one place, allowing faster Direct Memory Access (DMA) transfers to the GPU. You've already seen this with pin_memory=True in DataLoader.

Non-blocking Transfers: By default, .to(device) blocks Python execution until the transfer completes. Non-blocking transfers (tensor.to(device, non_blocking=True)) let the CPU continue working while data moves in the background — like starting the dishwasher and cooking dinner simultaneously instead of waiting.

Efficient Data Loading: Using DataLoader with num_workers > 0 and pin_memory=True means batches are prepared on CPU worker processes and pre-pinned, ready for immediate GPU transfer. This creates a pipeline where the next batch is prepared while the GPU processes the current one.

The Combined Effect

These techniques work together: workers prepare and pin batches in parallel, non-blocking transfers overlap data movement with computation, and pinned memory makes each transfer faster. The result: your GPU spends more time computing and less time waiting.

Key Takeaway: Minimize CPU-GPU transfer bottlenecks by using pinned memory for faster DMA transfers, non-blocking operations to overlap transfer with computation, and multi-worker DataLoaders to pipeline batch preparation.