Course contentsShow
← Machine Learning and Deep Learning
Lesson 2723 of 3,53859. Distributed Training: Data ParallelismFree lesson

Rank-Specific Logic and Master Process

Writing code that runs differently on rank 0 for logging, checkpointing, and validation.

Rank-Specific Logic and Master Process

What you'll learn: How to write conditional code that executes only on specific worker processes—especially rank 0—to avoid redundant logging, checkpointing, and validation in distributed training.

Why Rank-Specific Code Matters

In distributed data parallel training, multiple processes (workers) run identical copies of your training script simultaneously. Each process has a unique rank identifier: rank 0, rank 1, rank 2, etc.

Without rank-specific logic, every worker would independently:

  • Print training metrics (flooding your console)
  • Save checkpoints (creating race conditions or duplicate files)
  • Run validation loops (wasting computation)

The solution: Designate rank 0 as the "master process" responsible for orchestration tasks, while other ranks focus purely on training.

Implementing Rank-Specific Logic

You check the current process rank and conditionally execute code:

import torch.distributed as dist

# Get current process rank
rank = dist.get_rank()

# Only rank 0 performs logging and checkpointing
if rank == 0:
    print(f"Epoch {epoch}, Loss: {avg_loss:.4f}")
    torch.save(model.state_dict(), 'checkpoint.pth')
    
    # Run validation
    val_loss = validate(model, val_loader)
    print(f"Validation Loss: {val_loss:.4f}")

All ranks compute gradients and train, but only rank 0 handles I/O-heavy or once-per-epoch tasks.

Common Master Process Responsibilities

  • Logging: TensorBoard writes, progress bars, console output
  • Checkpointing: Saving model/optimizer states
  • Validation: Running validation loops (since metrics are the same across ranks)
  • Early stopping decisions: Checking convergence criteria

Key Takeaway: In distributed training, use rank-specific conditionals to designate rank 0 as the master process for logging, checkpointing, and validation—avoiding redundant operations and I/O bottlenecks across workers.