Launching Multi-Node Jobs with torchrun
What You'll Learn
You'll learn how to use PyTorch's torchrun utility to automatically spawn and coordinate training processes across multiple machines in a distributed cluster.
The Problem torchrun Solves
When training across multiple nodes (machines), each node needs to run multiple processes (typically one per GPU), and every process must know:
- Its global rank (unique ID across all nodes)
- The world size (total number of processes)
- How to find the rendezvous point (coordination server)
Manually managing these details is error-prone. torchrun automates this orchestration.
How torchrun Works
Think of torchrun as a smart process launcher. Instead of running python train.py directly, you run:
torchrun --nproc_per_node=4 --nnodes=2 --node_rank=0 \
--master_addr="192.168.1.1" --master_port=29500 \
train.py
Key parameters:
--nproc_per_node: Number of GPUs/processes per machine (e.g., 4 for a 4-GPU node)--nnodes: Total number of machines in your cluster--node_rank: Which machine this is (0 for master, 1, 2, ... for workers)--master_addrand--master_port: IP and port of the coordination node
On each machine, you run the same command with the appropriate --node_rank. torchrun automatically assigns global ranks, sets environment variables, and handles process failures.
Inside Your Training Script
Your train.py simply calls:
import torch.distributed as dist
dist.init_process_group(backend='nccl')
# torchrun has already set the required environment variables!
No manual rank calculation needed—torchrun handles it.
Key Takeaway: torchrun is PyTorch's standard tool for launching multi-node distributed training jobs, automatically managing process spawning, rank assignment, and coordination across machines with simple command-line flags.