Wrapping Models with DDP
You'll learn how to convert a standard PyTorch model into a distributed training-ready model using DistributedDataParallel.
The Core Transformation
Once you've initialized your process group (as covered in the previous lesson), the next step is wrapping your model with DDP. Think of this as putting a communication wrapper around your model—like adding a postal service layer that automatically ships gradients between workers and brings back averaged results.
The conversion is remarkably simple. You take your existing model and pass it to torch.nn.parallel.DistributedDataParallel:
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP
# Your standard model
model = MyNeuralNetwork().to(device)
# Wrap it with DDP
model = DDP(model, device_ids=[local_rank])
What Happens Under the Hood
When you wrap your model with DDP:
- DDP registers hooks on each parameter to intercept gradients during backpropagation
- During the backward pass, gradients are automatically all-reduced across all processes
- The wrapper ensures each process sees the same synchronized gradients
- Your model's forward pass remains unchanged—only the gradient communication is added
The device_ids parameter tells DDP which GPU this process should use. Each process wraps its own copy of the model on its assigned device.
Important Details
After wrapping, interact with your model normally—DDP handles synchronization transparently. The original model is accessible via model.module if needed for checkpointing or inspection.
Key Takeaway: Wrapping a model with torch.nn.parallel.DistributedDataParallel adds automatic gradient synchronization across workers with minimal code change—just one line transforms your single-GPU model into a distributed training setup.