Course contentsShow
← Machine Learning and Deep Learning
Lesson 2728 of 3,53859. Distributed Training: Data ParallelismFree lesson

DDP Debugging and Common Pitfalls

Troubleshooting hanging jobs, mismatched ranks, and batch size configuration errors.

DDP Debugging and Common Pitfalls

You'll learn how to identify and fix the most common issues that arise when using Distributed Data Parallel, including hanging jobs, rank mismatches, and batch size errors.

Why DDP Debugging Matters

Distributed training introduces complexity through multiple processes communicating across GPUs or nodes. When something goes wrong, jobs can hang indefinitely, crash mysteriously, or produce incorrect results—often with cryptic error messages or none at all.

Common Pitfalls and Solutions

Hanging Jobs

The most frustrating issue: your training simply stops progressing. This typically happens when:

  • Mismatched collective operations: If rank 0 calls all_reduce but rank 1 doesn't, they'll wait forever for each other
  • Inconsistent control flow: Using rank-specific if statements around DDP operations breaks synchronization
  • Timeout issues: Default timeout (30 minutes) may be too short for slow initialization

Solution: Ensure all ranks execute identical collective operations. Use torch.distributed.barrier() to synchronize before debugging prints.

Rank Mismatches

Occurs when processes have inconsistent views of the world:

  • Wrong RANK or WORLD_SIZE environment variables
  • Initialization with incompatible backend settings
  • Accessing wrong GPU device for a given rank

Solution: Always validate that rank matches the intended GPU via torch.cuda.set_device(rank) and print rank/world_size at startup.

Batch Size Configuration Errors

A subtle but critical mistake: forgetting that your effective batch size is per_gpu_batch_size × world_size. This affects:

  • Learning rate scaling (should scale proportionally)
  • Gradient accumulation steps
  • Memory consumption per GPU

Solution: Explicitly calculate and log your effective batch size. Adjust learning rate accordingly using linear scaling rules.

Key Takeaway: DDP failures often stem from synchronization mismatches between ranks or incorrect batch size assumptions; always ensure collective operations execute uniformly and validate your effective training configuration across all processes.