Gradient Synchronization Mechanics
What you'll learn: How DistributedDataParallel automatically averages gradients across all processes during the backward pass to keep model replicas in sync.
The Core Mechanism
When you call .backward() in a DDP training loop, something crucial happens behind the scenes: DDP automatically synchronizes gradients across all processes. Each worker computes gradients on its own mini-batch, but before the optimizer steps, DDP ensures every worker has the averaged gradient from all processes.
Here's how it works:
- Local backward pass: Each process computes gradients on its local batch independently
- Automatic all-reduce: DDP registers hooks on each parameter that trigger during backpropagation
- Gradient averaging: As soon as a parameter's gradient is ready, DDP launches an all-reduce operation to sum gradients across all workers
- Division by world size: The summed gradient is divided by the number of processes to get the average
- In-place replacement: Each worker's local gradient is replaced with this global average
Why This Matters
Think of it like a team solving math problems: each member solves their assigned problem (local batch), then everyone shares answers and computes the class average before anyone updates their notebook. This ensures all replicas stay perfectly synchronized—they all see the same gradient and take identical optimization steps.
The beauty of DDP is that this synchronization is overlapped with computation. While later layers are still backpropagating, earlier layers' gradients are already being communicated. This pipelining minimizes communication overhead.
Key Takeaway: DDP's automatic gradient synchronization ensures all model replicas remain identical by averaging gradients across processes during backward pass, using efficient all-reduce operations that overlap with computation to minimize overhead.