DDP Architecture and Communication Pattern
What You'll Learn
You'll learn how PyTorch's Distributed Data Parallel implements ring-allreduce to efficiently synchronize gradients across multiple processes during training.
The DDP Communication Architecture
DDP builds on the ring-allreduce concept you learned earlier, but adds an intelligent process-level architecture. Each training process gets its own replica of the model and operates independently on different data batches. The magic happens during the backward pass.
How Communication Actually Happens
When you call loss.backward() in DDP, gradients don't wait until all layers finish computing. Instead, DDP uses gradient bucketing: as soon as a bucket of gradients is ready (computed in reverse layer order), it immediately launches an all-reduce operation in the background.
Think of it like an assembly line: while early layers are still computing gradients, later layers' gradients are already being averaged across processes. This overlaps computation with communication, hiding much of the synchronization cost.
The Ring Topology
DDP arranges processes in a logical ring (just like ring-allreduce). Each process communicates only with its neighbors, passing gradient chunks around the ring. After one full ring traversal, every process has the averaged gradient for each bucket.
Process Groups and Backends
DDP relies on a process group managed by a backend (like NCCL for GPUs or Gloo for CPUs). The backend handles the low-level communication primitives. Each process knows its rank and the total world size, enabling coordinated but independent execution.
The architecture is synchronous by default: all processes must complete their all-reduce before proceeding to the optimizer step, ensuring model replicas stay identical.
Key Takeaway: DDP's architecture uses bucketed gradient all-reduce with a ring communication pattern, overlapping backward pass computation with inter-process communication to minimize synchronization overhead while keeping all model replicas synchronized.