Process Groups and Initialization
What you'll learn: How to set up communication channels between distributed workers and understand their identities in the training cluster.
The Core Idea
Before multiple processes can work together in Distributed Data Parallel training, they need to know about each other and establish communication channels. Think of it like setting up a conference call—everyone needs a dial-in number (the rendezvous method), a participant ID (rank), and knowledge of how many people are joining (world_size).
Setting Up Process Groups
init_process_group is the initialization function that coordinates all workers. Each process calls this function independently to join the collective:
import torch.distributed as dist
dist.init_process_group(
backend='nccl', # Communication backend (nccl for GPUs)
init_method='env://', # How to discover other processes
world_size=4, # Total number of processes
rank=0 # This process's unique ID (0 to world_size-1)
)
Understanding World Size and Rank
world_size: The total number of processes participating in training (e.g., 4 GPUs = world_size of 4)rank: The unique identifier for each process, from 0 toworld_size - 1. Rank 0 is typically the "master" process that handles logging and checkpointing
If you have 4 GPUs, you launch 4 separate Python processes, each with its own rank (0, 1, 2, 3) but all sharing the same world_size (4).
Key Takeaway: init_process_group establishes the communication infrastructure for DDP by assigning each process a unique rank within a world of a specified size, enabling coordinated gradient synchronization across all workers.