Course contentsShow
← Machine Learning and Deep Learning
Lesson 2715 of 3,53859. Distributed Training: Data ParallelismFree lesson

What is Distributed Data Parallel (DDP)?

Overview of DDP's role in multi-GPU training and how it differs from basic DataParallel.

What is Distributed Data Parallel (DDP)?

You'll learn: How PyTorch's DistributedDataParallel (DDP) orchestrates efficient multi-GPU training and why it outperforms the simpler DataParallel approach.

The Core Idea

Distributed Data Parallel (DDP) is PyTorch's recommended approach for training models across multiple GPUs, whether on a single machine or across multiple nodes. Unlike the basic DataParallel wrapper you learned about earlier, DDP creates a separate process for each GPU, giving each process its own copy of the model and optimizer.

Think of DataParallel as a single manager delegating work to multiple workers but coordinating everything centrally—the main GPU becomes a bottleneck. DDP is more like multiple independent managers working in parallel, only syncing when absolutely necessary.

How DDP Differs from DataParallel

Process vs Thread Model: DataParallel uses Python multithreading from one process, suffering from the Global Interpreter Lock (GIL). DDP uses multiprocessing—one process per GPU—eliminating GIL overhead and enabling true parallelism.

Gradient Synchronization: DataParallel gathers all gradients to GPU 0, averages them there, then broadcasts updated parameters back—creating a communication bottleneck. DDP performs all-reduce operations directly across all processes using optimized backends (like NCCL), distributing communication more efficiently.

Model Replication: In DataParallel, the model lives primarily on GPU 0 and gets scattered each forward pass. DDP maintains a complete model replica in each process from the start, only synchronizing gradients during backward passes.

Scalability: DDP scales to multi-node training seamlessly, while DataParallel is limited to a single machine.

Why DDP Matters

DDP typically achieves 2-3x faster training than DataParallel on the same hardware by eliminating central bottlenecks and leveraging efficient collective communication primitives you learned about with ring all-reduce.

Key Takeaway: DDP is PyTorch's efficient multi-GPU solution that uses separate processes per GPU and optimized gradient synchronization to eliminate the bottlenecks inherent in DataParallel's single-process design.