Course contentsShow
← Machine Learning and Deep Learning
Lesson 2703 of 3,53859. Distributed Training: Data ParallelismFree lesson

Why Distributed Training Is Necessary

Understanding memory constraints, training time bottlenecks, and when single-GPU training becomes insufficient.

Why Distributed Training Is Necessary

You'll learn: When and why training on a single GPU becomes a bottleneck that demands distributed solutions.

The Single-GPU Wall

As deep learning models grow larger—think GPT-class transformers with billions of parameters or high-resolution vision models—a single GPU eventually hits hard limits. These constraints fall into two categories: memory and time.

Memory Constraints

Modern GPUs have finite memory (typically 16–80 GB). Your training process must fit:

  • Model parameters (weights, biases)
  • Gradients (same size as parameters)
  • Optimizer states (momentum, variance buffers—often 2× parameter size for Adam)
  • Activation maps from forward passes (grows with batch size and model depth)

A large language model with 7 billion float32 parameters alone requires ~28 GB just for weights. Add gradients, optimizer states, and activations, and you've exceeded most consumer GPUs before training even begins.

Training Time Bottlenecks

Even if your model fits in memory, training might take weeks or months on a single device. A batch size of 32 on one GPU might process 1,000 samples per hour, but scaling to millions of training examples becomes impractical for research iteration or production deadlines.

When Single-GPU Training Becomes Insufficient

You need distributed training when:

  • Your model literally doesn't fit in GPU memory
  • Training time exceeds your iteration budget (hours → days → weeks)
  • You want to use larger batch sizes for improved gradient estimates or faster convergence
  • You're scaling experiments that demand higher throughput

Key Takeaway: Distributed training becomes necessary when memory capacity or training duration on a single device blocks progress—either the model won't fit, or waiting weeks for results stalls your research and deployment cycles.