Distributed Process Groups and Ranks
You'll learn how distributed training organizes multiple processes using ranks, world size, and process groups to coordinate computation across devices.
The Core Idea
When you run distributed training, each GPU or compute device runs its own Python process. These processes need a way to identify themselves and communicate. The rank is a unique ID for each process (0, 1, 2, ...), the world size is the total number of processes, and a process group is the communication channel connecting them all.
Think of it like a team project: if 8 people are working together, each person gets an ID badge (rank 0-7), everyone knows there are 8 team members (world size = 8), and they all join the same chat group (process group) to share updates.
Key Components
World Size
The total number of participating processes. With 4 GPUs across 2 nodes (2 GPUs each), your world size is 4.
Rank Assignment
Each process gets a unique integer from 0 to world_size - 1. Rank 0 typically becomes the master node (or coordinator), handling tasks like logging, saving checkpoints, and validation.
Process Groups
The communication backend (like NCCL for GPUs or Gloo for CPUs) that enables all processes to exchange data. All ranks join the same default group during initialization.
Worker Coordination
Ranks synchronize through collective operations you've seen before: all_reduce for gradients, broadcast for initial weights, and barrier for synchronization points.
Key Takeaway: Process groups with rank assignment create the organizational structure for distributed training—each worker knows its ID (rank), the team size (world size), and how to communicate (process group), enabling coordinated parallel computation across multiple devices.