Distributed Samplers for Multi-GPU Training
What you'll learn: How to partition your dataset across multiple GPUs so each device processes unique data without overlap or missing samples.
The Problem with Naive Multi-GPU Data Loading
When training on multiple GPUs, each device runs its own copy of the model and processes batches independently. If you use a standard DataLoader on each GPU without coordination, every GPU will see the same data in the same order—essentially wasting your parallel hardware by doing redundant work.
What is a DistributedSampler?
A DistributedSampler is PyTorch's solution to intelligently divide your dataset across GPUs. Think of it like dealing cards: if you have 4 players (GPUs) and 52 cards (samples), each player should get exactly 13 unique cards, with no duplicates and no cards left out.
The sampler automatically:
- Calculates how many samples each GPU should handle
- Assigns non-overlapping subsets to each device
- Ensures the entire dataset is covered exactly once per epoch
How It Works
from torch.utils.data.distributed import DistributedSampler
# Assuming you've initialized distributed training
sampler = DistributedSampler(
dataset,
num_replicas=world_size, # Total number of GPUs
rank=rank # This GPU's ID (0 to world_size-1)
)
dataloader = DataLoader(
dataset,
batch_size=32,
sampler=sampler,
shuffle=False # Sampler handles shuffling
)
Each GPU gets its own DataLoader with the same sampler configuration but a different rank, ensuring they see different data slices.
Key Takeaway: DistributedSampler partitions your dataset across GPUs by giving each device a unique, non-overlapping subset of indices, enabling efficient parallel training without data duplication.