Saving and Loading DDP Checkpoints
What you'll learn: How to properly save and restore model state in distributed training without duplicating work or causing corruption.
Why Checkpoints Need Special Care in DDP
When training with DistributedDataParallel, each process holds a replica of the model. If every process tried to save the checkpoint simultaneously, you'd get race conditions, wasted I/O bandwidth, and potentially corrupted files. The solution: only the master process (rank 0) should save checkpoints.
Saving from Rank 0
Since DDP wraps your original model, you need to extract the underlying model state before saving. Use rank-specific logic:
if torch.distributed.get_rank() == 0:
# Access the original model through .module
checkpoint = {
'model_state_dict': model.module.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'epoch': epoch,
}
torch.save(checkpoint, 'checkpoint.pth')
The key is model.module — this unwraps the DDP wrapper to get your original model's parameters.
Loading Across All Ranks
When loading, every process must load the checkpoint to keep their model replicas synchronized. Load first, then wrap with DDP:
checkpoint = torch.load('checkpoint.pth', map_location=device)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
# Now wrap with DDP
model = DDP(model, device_ids=[local_rank])
Analogy: Think of it like a choir where only the conductor writes down the song arrangement (rank 0 saves), but every singer needs to read it (all ranks load) before performing together.
Key Takeaway: Save checkpoints only from rank 0 using model.module.state_dict(), but load checkpoints on all ranks before wrapping with DDP to ensure synchronized training resumption.