Course contentsShow
← Machine Learning and Deep Learning
Lesson 2794 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Distributed Process Groups and Ranks

Setting up process groups: world size, rank assignment, master node, and worker coordination.

Distributed Process Groups and Ranks

You'll learn how distributed training organizes multiple processes using ranks, world size, and process groups to coordinate computation across devices.

The Core Idea

When you run distributed training, each GPU or compute device runs its own Python process. These processes need a way to identify themselves and communicate. The rank is a unique ID for each process (0, 1, 2, ...), the world size is the total number of processes, and a process group is the communication channel connecting them all.

Think of it like a team project: if 8 people are working together, each person gets an ID badge (rank 0-7), everyone knows there are 8 team members (world size = 8), and they all join the same chat group (process group) to share updates.

Key Components

World Size

The total number of participating processes. With 4 GPUs across 2 nodes (2 GPUs each), your world size is 4.

Rank Assignment

Each process gets a unique integer from 0 to world_size - 1. Rank 0 typically becomes the master node (or coordinator), handling tasks like logging, saving checkpoints, and validation.

Process Groups

The communication backend (like NCCL for GPUs or Gloo for CPUs) that enables all processes to exchange data. All ranks join the same default group during initialization.

Worker Coordination

Ranks synchronize through collective operations you've seen before: all_reduce for gradients, broadcast for initial weights, and barrier for synchronization points.

Key Takeaway: Process groups with rank assignment create the organizational structure for distributed training—each worker knows its ID (rank), the team size (world size), and how to communicate (process group), enabling coordinated parallel computation across multiple devices.