Single-Node Multi-GPU Training
What you'll learn: How to implement DistributedDataParallel (DDP) training on a single machine with multiple GPUs using PyTorch's torch.distributed.launch utility.
The Core Idea
You've learned DDP's architecture, process groups, model wrapping, and gradient synchronization. Now it's time to actually run it. Single-node multi-GPU training means using all GPUs on one physical machine to train your model in parallel. PyTorch provides torch.distributed.launch (and its newer equivalent torchrun) as a launcher that spawns one process per GPU automatically.
How It Works
Instead of manually starting multiple processes, the launcher:
- Spawns N processes (one per GPU you specify)
- Sets environment variables like
RANK,WORLD_SIZE, andLOCAL_RANKfor each process - Assigns each process to a different GPU using these variables
Your training script remains mostly the same—you just initialize the process group, wrap your model with DDP, and use a distributed sampler. The launcher handles the multi-process orchestration.
Typical command:
python -m torch.distributed.launch --nproc_per_node=4 train.py
This launches 4 processes (for 4 GPUs) on your local machine. Each process runs train.py with its own unique rank.
In your script:
torch.cuda.set_device(local_rank)
model = model.to(local_rank)
model = DDP(model, device_ids=[local_rank])
The local_rank (GPU ID on this node) comes from the environment variables set by the launcher.
Key Takeaway: Single-node multi-GPU training with DDP uses torch.distributed.launch to automatically spawn one process per GPU, eliminating manual process management while enabling efficient parallel training on all available GPUs.