Rank-Specific Logic and Master Process
What you'll learn: How to write conditional code that executes only on specific worker processes—especially rank 0—to avoid redundant logging, checkpointing, and validation in distributed training.
Why Rank-Specific Code Matters
In distributed data parallel training, multiple processes (workers) run identical copies of your training script simultaneously. Each process has a unique rank identifier: rank 0, rank 1, rank 2, etc.
Without rank-specific logic, every worker would independently:
- Print training metrics (flooding your console)
- Save checkpoints (creating race conditions or duplicate files)
- Run validation loops (wasting computation)
The solution: Designate rank 0 as the "master process" responsible for orchestration tasks, while other ranks focus purely on training.
Implementing Rank-Specific Logic
You check the current process rank and conditionally execute code:
import torch.distributed as dist
# Get current process rank
rank = dist.get_rank()
# Only rank 0 performs logging and checkpointing
if rank == 0:
print(f"Epoch {epoch}, Loss: {avg_loss:.4f}")
torch.save(model.state_dict(), 'checkpoint.pth')
# Run validation
val_loss = validate(model, val_loader)
print(f"Validation Loss: {val_loss:.4f}")
All ranks compute gradients and train, but only rank 0 handles I/O-heavy or once-per-epoch tasks.
Common Master Process Responsibilities
- Logging: TensorBoard writes, progress bars, console output
- Checkpointing: Saving model/optimizer states
- Validation: Running validation loops (since metrics are the same across ranks)
- Early stopping decisions: Checking convergence criteria
Key Takeaway: In distributed training, use rank-specific conditionals to designate rank 0 as the master process for logging, checkpointing, and validation—avoiding redundant operations and I/O bottlenecks across workers.