Course contentsShow
← Machine Learning and Deep Learning
Lesson 2719 of 3,53859. Distributed Training: Data ParallelismFree lesson

Distributed Samplers for Data Loading

Using DistributedSampler to ensure each process gets unique data without overlap.

Distributed Samplers for Data Loading

What You'll Learn

You'll learn how to use DistributedSampler to partition your dataset across multiple processes so each worker trains on unique, non-overlapping data batches.

The Problem: Data Overlap in Distributed Training

When running data parallel training across multiple GPUs or nodes, each process needs its own DataLoader. Without coordination, every process would iterate through the entire dataset identically—meaning they'd all see the same batch 0, same batch 1, etc. This defeats the purpose of distributed training since you're effectively repeating work across workers.

How DistributedSampler Solves It

Think of it like dealing cards: If you have 4 players (processes) and 100 cards (data samples), you deal card 0 to player 0, card 1 to player 1, card 2 to player 2, card 3 to player 3, then card 4 back to player 0, and so on. Each player gets a unique subset.

DistributedSampler does exactly this—it partitions indices based on:

  • rank: Which process this is (0, 1, 2...)
  • world_size: Total number of processes

Each process gets len(dataset) // world_size samples, ensuring no overlap. The sampler also handles shuffling consistently across epochs when needed.

Practical Usage

You create a sampler before your DataLoader:

from torch.utils.data.distributed import DistributedSampler

sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)
dataloader = DataLoader(dataset, batch_size=32, sampler=sampler)

Remember: when using DistributedSampler, set shuffle=False in DataLoader (the sampler handles shuffling).

Key Takeaway: DistributedSampler partitions your dataset by process rank, ensuring each worker in distributed training sees unique data without duplication or overlap.