Course contentsShow
← Machine Learning and Deep Learning
Lesson 2713 of 3,53859. Distributed Training: Data ParallelismFree lesson

DataParallel vs DistributedDataParallel in PyTorch

Differences between torch.nn.DataParallel (single-process) and DDP (multi-process) and why DDP is preferred.

DataParallel vs DistributedDataParallel in PyTorch

You'll learn: Why PyTorch's DistributedDataParallel (DDP) outperforms the simpler DataParallel (DP) for multi-GPU training.

The Core Difference: Single-Process vs Multi-Process

PyTorch offers two built-in methods for data parallelism:

torch.nn.DataParallel (DP) runs in a single process with multiple threads. It replicates your model to all GPUs, but the main thread on GPU 0 handles all the coordination: scattering inputs, gathering outputs, averaging gradients, and updating parameters. This creates a bottleneck—GPU 0 becomes overworked while others wait.

torch.nn.parallel.DistributedDataParallel (DDP) launches separate processes, one per GPU. Each process owns its model replica and independently performs forward/backward passes. Gradients are synchronized across processes using efficient all-reduce operations (like ring all-reduce you learned earlier), with no single "master" GPU bottleneck.

Why DDP Is Preferred

Communication efficiency: DP uses inefficient scatter/gather operations through a single GPU. DDP uses optimized collective communications (all-reduce) that distribute work evenly.

Scaling: DP's single-process design limits it to one machine. DDP works seamlessly across multiple nodes in a cluster.

Performance: DDP eliminates the GPU 0 bottleneck. All GPUs work equally, and gradient synchronization happens during the backward pass (overlapping computation and communication).

Python GIL: DP's multithreading can hit Python's Global Interpreter Lock limitations. DDP's multiprocessing avoids this entirely.

Think of DP as one manager coordinating multiple workers but doing all the paperwork themselves—they become the bottleneck. DDP is like workers who coordinate peer-to-peer, sharing updates directly without a middleman.

Key Takeaway: Use DistributedDataParallel for serious multi-GPU training—it avoids bottlenecks through multi-process design and efficient gradient all-reduce, making it faster and more scalable than single-process DataParallel.