Course contentsShow
← Machine Learning and Deep Learning
Lesson 823 of 3,53819. PyTorch FundamentalsFree lesson

Distributed Samplers for Multi-GPU Training

Implement DistributedSampler to partition data across multiple GPUs without duplication or gaps.

Distributed Samplers for Multi-GPU Training

What you'll learn: How to partition your dataset across multiple GPUs so each device processes unique data without overlap or missing samples.

The Problem with Naive Multi-GPU Data Loading

When training on multiple GPUs, each device runs its own copy of the model and processes batches independently. If you use a standard DataLoader on each GPU without coordination, every GPU will see the same data in the same order—essentially wasting your parallel hardware by doing redundant work.

What is a DistributedSampler?

A DistributedSampler is PyTorch's solution to intelligently divide your dataset across GPUs. Think of it like dealing cards: if you have 4 players (GPUs) and 52 cards (samples), each player should get exactly 13 unique cards, with no duplicates and no cards left out.

The sampler automatically:

  • Calculates how many samples each GPU should handle
  • Assigns non-overlapping subsets to each device
  • Ensures the entire dataset is covered exactly once per epoch

How It Works

from torch.utils.data.distributed import DistributedSampler

# Assuming you've initialized distributed training
sampler = DistributedSampler(
    dataset,
    num_replicas=world_size,  # Total number of GPUs
    rank=rank                  # This GPU's ID (0 to world_size-1)
)

dataloader = DataLoader(
    dataset,
    batch_size=32,
    sampler=sampler,
    shuffle=False  # Sampler handles shuffling
)

Each GPU gets its own DataLoader with the same sampler configuration but a different rank, ensuring they see different data slices.

Key Takeaway: DistributedSampler partitions your dataset across GPUs by giving each device a unique, non-overlapping subset of indices, enabling efficient parallel training without data duplication.