Course contentsShow
← Machine Learning and Deep Learning
Lesson 849 of 3,53819. PyTorch FundamentalsFree lesson

Multi-GPU Basics: DataParallel

Introduce nn.DataParallel for simple multi-GPU training by replicating models across available GPUs.

Multi-GPU Basics: DataParallel

What You'll Learn

You'll learn how to use nn.DataParallel to automatically distribute your model across multiple GPUs for faster training.

The Core Idea

Think of DataParallel like a copy machine for your model. It creates identical copies of your model on each available GPU, splits each batch of data across those GPUs, runs forward passes in parallel, then gathers the results and averages the gradients before updating the model.

When you have multiple GPUs on one machine, you can train significantly faster by processing different parts of your batch simultaneously. DataParallel is PyTorch's simplest tool for this—it wraps your model and handles the parallelization automatically.

How DataParallel Works

The workflow happens automatically:

  1. Replicate: Your model is copied to all available GPUs
  2. Scatter: Each mini-batch is split across GPUs (if batch size is 32 and you have 4 GPUs, each gets 8 samples)
  3. Parallel Forward: Each GPU processes its portion independently
  4. Gather: Outputs are collected back to the primary GPU
  5. Loss & Backward: Gradients are computed and averaged across GPUs
  6. Update: Model parameters are synchronized

Using DataParallel

import torch
import torch.nn as nn

# Your model (already defined)
model = YourModel()

# Wrap with DataParallel (uses all available GPUs by default)
model = nn.DataParallel(model)
model = model.to('cuda')

# Training proceeds normally
outputs = model(inputs)  # DataParallel handles the rest

You can also specify which GPUs to use:

model = nn.DataParallel(model, device_ids=[0, 1, 2])  # Use GPUs 0, 1, 2

Key Takeaway: nn.DataParallel provides an easy way to split batches across multiple GPUs on a single machine with minimal code changes—just wrap your model before moving it to GPU.