num_workers: Multiprocess Data Loading
What you'll learn: How to use multiple worker processes to load data in parallel and speed up training by preventing data loading from becoming your bottleneck.
The Data Loading Bottleneck
Imagine you're at a restaurant where the chef (GPU) can cook extremely fast, but there's only one waiter (main process) fetching ingredients from the kitchen. The chef ends up waiting, idle, while the waiter runs back and forth. That's what happens when your model trains faster than your data can be loaded from disk.
What is num_workers?
The num_workers parameter in PyTorch's DataLoader controls how many separate worker processes load data in parallel. When set to a value greater than 0, multiple processes simultaneously read from disk, apply transformations, and prepare batches—all while your GPU trains on the current batch.
How It Works
from torch.utils.data import DataLoader
# Single-process loading (default)
dataloader_slow = DataLoader(dataset, batch_size=32, num_workers=0)
# Multi-process loading with 4 workers
dataloader_fast = DataLoader(dataset, batch_size=32, num_workers=4)
With num_workers=4, four separate processes work in parallel to prepare your next batches. While the GPU processes batch N, the workers are already loading and preprocessing batches N+1, N+2, and beyond.
Choosing the Right Number
- num_workers=0: Single process (main thread loads data)
- num_workers=2-8: Common sweet spot for most machines
- Rule of thumb: Start with
num_workers=4and adjust based on your CPU cores and disk speed
Too many workers can actually slow things down due to process overhead and memory duplication.
The Performance Impact
When data loading is your bottleneck, adding workers can dramatically reduce training time—sometimes cutting it in half or more—by keeping your GPU continuously fed with data instead of waiting idle.
Key Takeaway: Use num_workers in your DataLoader to parallelize data loading across multiple processes, eliminating idle GPU time and significantly speeding up training when I/O is your bottleneck.