Course contentsShow
← Machine Learning and Deep Learning
Lesson 2717 of 3,53859. Distributed Training: Data ParallelismFree lesson

Process Groups and Initialization

Setting up process groups with init_process_group and understanding world_size and rank.

Process Groups and Initialization

What you'll learn: How to set up communication channels between distributed workers and understand their identities in the training cluster.

The Core Idea

Before multiple processes can work together in Distributed Data Parallel training, they need to know about each other and establish communication channels. Think of it like setting up a conference call—everyone needs a dial-in number (the rendezvous method), a participant ID (rank), and knowledge of how many people are joining (world_size).

Setting Up Process Groups

init_process_group is the initialization function that coordinates all workers. Each process calls this function independently to join the collective:

import torch.distributed as dist

dist.init_process_group(
    backend='nccl',           # Communication backend (nccl for GPUs)
    init_method='env://',     # How to discover other processes
    world_size=4,             # Total number of processes
    rank=0                    # This process's unique ID (0 to world_size-1)
)

Understanding World Size and Rank

  • world_size: The total number of processes participating in training (e.g., 4 GPUs = world_size of 4)
  • rank: The unique identifier for each process, from 0 to world_size - 1. Rank 0 is typically the "master" process that handles logging and checkpointing

If you have 4 GPUs, you launch 4 separate Python processes, each with its own rank (0, 1, 2, 3) but all sharing the same world_size (4).

Key Takeaway: init_process_group establishes the communication infrastructure for DDP by assigning each process a unique rank within a world of a specified size, enabling coordinated gradient synchronization across all workers.