Course contentsShow
← Machine Learning and Deep Learning
Lesson 2795 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Launching Multi-Node Jobs with torchrun

Using PyTorch's torchrun (formerly torch.distributed.launch) to spawn processes across multiple nodes.

Launching Multi-Node Jobs with torchrun

What You'll Learn

You'll learn how to use PyTorch's torchrun utility to automatically spawn and coordinate training processes across multiple machines in a distributed cluster.

The Problem torchrun Solves

When training across multiple nodes (machines), each node needs to run multiple processes (typically one per GPU), and every process must know:

  • Its global rank (unique ID across all nodes)
  • The world size (total number of processes)
  • How to find the rendezvous point (coordination server)

Manually managing these details is error-prone. torchrun automates this orchestration.

How torchrun Works

Think of torchrun as a smart process launcher. Instead of running python train.py directly, you run:

torchrun --nproc_per_node=4 --nnodes=2 --node_rank=0 \
         --master_addr="192.168.1.1" --master_port=29500 \
         train.py

Key parameters:

  • --nproc_per_node: Number of GPUs/processes per machine (e.g., 4 for a 4-GPU node)
  • --nnodes: Total number of machines in your cluster
  • --node_rank: Which machine this is (0 for master, 1, 2, ... for workers)
  • --master_addr and --master_port: IP and port of the coordination node

On each machine, you run the same command with the appropriate --node_rank. torchrun automatically assigns global ranks, sets environment variables, and handles process failures.

Inside Your Training Script

Your train.py simply calls:

import torch.distributed as dist

dist.init_process_group(backend='nccl')
# torchrun has already set the required environment variables!

No manual rank calculation needed—torchrun handles it.

Key Takeaway: torchrun is PyTorch's standard tool for launching multi-node distributed training jobs, automatically managing process spawning, rank assignment, and coordination across machines with simple command-line flags.