Course contentsShow
← Machine Learning and Deep Learning
Lesson 683 of 3,53817. Optimization for Deep LearningFree lesson

From Batch GD to Stochastic GD

Why full-batch gradient descent is impractical for deep learning and how stochastic sampling enables scalability.

From Batch GD to Stochastic GD

What you'll learn: Why deep learning can't rely on full-batch gradient descent and how random sampling unlocks practical training at scale.

The Full-Batch Problem

In batch gradient descent (BGD), you compute the loss and gradients using every single training example before taking one optimization step. For a dataset with millions of images, this means:

  1. Forward propagating all examples through your network
  2. Computing the average loss across all data
  3. Backpropagating through all examples
  4. Finally updating weights once

The problem? This is computationally prohibitive for deep learning. Imagine processing 10 million images just to take a single weight update step—training would take forever, and you'd need massive memory to hold all activations.

The Stochastic Solution

Stochastic Gradient Descent (SGD) solves this by using one random example at a time:

  • Pick a single training sample
  • Compute its loss and gradients
  • Update weights immediately
  • Repeat with another random sample

Think of it like learning to cook: instead of studying every recipe in existence before making your first dish (batch GD), you try one recipe, learn from mistakes, adjust your technique, then try another (stochastic GD).

Why This Enables Scalability

Stochastic sampling means each weight update is cheap and immediate. You don't need to store millions of intermediate activations in memory. Training becomes an iterative process of rapid, noisy updates that still converge toward good solutions—the randomness actually helps escape shallow local minima!

Key Takeaway: Full-batch gradient descent is impractical for deep learning due to computational and memory constraints; stochastic gradient descent enables scalability by computing gradients from single random examples, trading exact gradients for fast, frequent updates.