Course contentsShow
← Machine Learning and Deep Learning
Lesson 2704 of 3,53859. Distributed Training: Data ParallelismFree lesson

Data Parallelism Overview

Core concept: replicating the model across devices and splitting data batches for parallel gradient computation.

Data Parallelism Overview

What you'll learn: How to speed up training by copying your model to multiple devices and processing different data batches simultaneously.

The Core Idea

Data parallelism is the most common distributed training strategy. Instead of training on one device with small batches, you replicate your entire model across multiple GPUs or machines, then split each training batch into smaller chunks—one chunk per device. Each device computes gradients on its portion independently, then all devices synchronize by averaging their gradients before updating the model weights.

Think of it like grading exams: instead of one teacher grading 100 papers sequentially, you have 4 teachers each grade 25 papers simultaneously, then they meet to discuss and agree on the grading standards together.

How It Works

  1. Model Replication: Each GPU gets an identical copy of the model with the same weights
  2. Data Distribution: A batch of size 256 might be split into 4 sub-batches of 64, one per GPU
  3. Parallel Forward/Backward: Each GPU independently runs forward and backward passes on its data chunk
  4. Gradient Synchronization: All GPUs communicate to average their computed gradients
  5. Synchronized Update: Each model replica updates using the averaged gradient

This approach is particularly effective when you're constrained by batch size due to memory limits—you can achieve an effective batch size of 256 across 4 GPUs even though each GPU only holds 64 examples.

When to Use Data Parallelism

Data parallelism shines when your model fits in a single device's memory but you want faster training or larger effective batch sizes. It's the go-to strategy for image classification, most NLP tasks, and any scenario where data is abundant and processing it faster is the bottleneck.

Key Takeaway: Data parallelism speeds up training by replicating your model across devices and processing different data subsets in parallel, synchronizing gradients to maintain training consistency.