Course contentsShow
← Machine Learning and Deep Learning
Lesson 2705 of 3,53859. Distributed Training: Data ParallelismFree lesson

The Data Parallel Training Loop

Step-by-step breakdown: forward pass on different data, backward pass, gradient synchronization, and parameter update.

The Data Parallel Training Loop

What you'll learn: How data parallelism orchestrates forward passes, backward passes, gradient synchronization, and parameter updates across multiple devices.

The Core Idea

In data parallel training, each worker (GPU/device) holds an identical copy of the model but processes a different subset of the training batch. The training loop follows a carefully coordinated sequence to keep all model replicas in sync while maximizing parallel computation.

Step-by-Step Breakdown

1. Forward Pass on Different Data

Each worker receives a unique mini-batch shard and independently computes predictions. If you have 4 GPUs and a batch of 128 examples, each GPU processes 32 examples through its model copy.

2. Backward Pass

Each worker computes gradients based on its local data shard. These gradients are calculated independently and initially exist only on their respective devices—they're not yet synchronized.

3. Gradient Synchronization

This is the critical coordination step. All workers communicate to average (or sum) their gradients across devices. Think of it like a team where each member solves part of a problem, then they meet to combine their findings into a unified solution. This typically uses an all-reduce operation that efficiently shares gradients across all workers.

4. Parameter Update

Once synchronized gradients are available, each worker applies the optimizer step using the averaged gradients. Because all workers started with identical parameters and now use identical gradients, they remain perfectly synchronized after the update.

The Key Insight

The beauty of this loop is that computation (forward/backward) happens in parallel, while communication (synchronization) happens only once per training step—maximizing efficiency while ensuring model consistency.

Key Takeaway: The data parallel training loop splits computation across workers but synchronizes gradients before updates, ensuring all model replicas stay identical while processing more data per step.