Course contentsShow
← Machine Learning and Deep Learning
Lesson 2727 of 3,53859. Distributed Training: Data ParallelismFree lesson

DDP Performance Optimization

Tuning bucket sizes, gradient_as_bucket_view, and find_unused_parameters for efficiency.

DDP Performance Optimization

What you'll learn: How to fine-tune DDP's internal parameters—bucket sizes, gradient_as_bucket_view, and find_unused_parameters—to minimize communication overhead and maximize training speed.

Why DDP Needs Tuning

By default, DDP works well, but its performance can be significantly improved by adjusting how gradients are bucketed and communicated. Think of it like packing boxes for shipping: the right box sizes and packing strategy can dramatically reduce shipping time and cost.

Core Optimization Levers

Bucket Size (bucket_cap_mb)

DDP groups gradients into "buckets" before all-reduce operations. Larger buckets mean fewer communication calls but higher latency per call. Smaller buckets enable more overlap between computation and communication but create overhead from many small transfers.

Default: 25 MB. Try increasing to 100–200 MB for models with many parameters, or decreasing to 10 MB for faster overlap on high-bandwidth networks.

gradient_as_bucket_view=True

When enabled, DDP avoids copying gradients into separate bucket storage—instead, it creates views directly into the gradient tensors. This eliminates memory copies and reduces peak memory usage.

Use it: Almost always enable this for free performance gains (default in recent PyTorch versions).

find_unused_parameters=False

If your model has dynamic computation graphs (some parameters unused in certain forward passes), DDP must track which gradients to synchronize. This adds overhead.

Rule: Keep this False (default) unless you have control flow that conditionally uses layers. Unnecessary tracking slows down every iteration.

Practical Impact

Proper tuning can reduce communication time by 20–40%, especially on multi-node setups where network bandwidth is the bottleneck.

Key Takeaway: Optimize DDP by tuning bucket sizes for your network bandwidth, enabling gradient_as_bucket_view to eliminate copies, and keeping find_unused_parameters=False unless dynamic graphs require it.