Distributed Samplers for Data Loading
What You'll Learn
You'll learn how to use DistributedSampler to partition your dataset across multiple processes so each worker trains on unique, non-overlapping data batches.
The Problem: Data Overlap in Distributed Training
When running data parallel training across multiple GPUs or nodes, each process needs its own DataLoader. Without coordination, every process would iterate through the entire dataset identically—meaning they'd all see the same batch 0, same batch 1, etc. This defeats the purpose of distributed training since you're effectively repeating work across workers.
How DistributedSampler Solves It
Think of it like dealing cards: If you have 4 players (processes) and 100 cards (data samples), you deal card 0 to player 0, card 1 to player 1, card 2 to player 2, card 3 to player 3, then card 4 back to player 0, and so on. Each player gets a unique subset.
DistributedSampler does exactly this—it partitions indices based on:
rank: Which process this is (0, 1, 2...)world_size: Total number of processes
Each process gets len(dataset) // world_size samples, ensuring no overlap. The sampler also handles shuffling consistently across epochs when needed.
Practical Usage
You create a sampler before your DataLoader:
from torch.utils.data.distributed import DistributedSampler
sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)
dataloader = DataLoader(dataset, batch_size=32, sampler=sampler)
Remember: when using DistributedSampler, set shuffle=False in DataLoader (the sampler handles shuffling).
Key Takeaway: DistributedSampler partitions your dataset by process rank, ensuring each worker in distributed training sees unique data without duplication or overlap.