Course contentsShow
← Machine Learning and Deep Learning
Lesson 105 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Stochastic Gradient Descent Basics

Using noisy gradient estimates from mini-batches to scale optimization to large datasets.

Stochastic Gradient Descent Basics

What you'll learn: How to speed up optimization on massive datasets by using random mini-batches instead of the entire dataset at each step.

The Core Problem

You've learned that Gradient Descent computes the gradient using all your training data at every iteration. Imagine you have 1 million examples—computing the full gradient means processing all 1 million points just to take one step! This becomes painfully slow.

The Stochastic Solution

Stochastic Gradient Descent (SGD) takes a smarter approach: instead of computing the exact gradient from all data, it estimates the gradient using a small random subset called a mini-batch (often 32, 64, or 256 examples).

Think of it like a political poll. You don't need to ask all 300 million Americans their opinion—a random sample of 1,000 people gives you a noisy but useful estimate of the population's view. Similarly, a mini-batch gives you a noisy gradient estimate that on average points in the right direction.

How It Works

At each iteration:

  1. Randomly select a mini-batch from your dataset
  2. Compute the gradient using only those examples
  3. Take a step in that (approximate) direction
  4. Repeat with a new random mini-batch

The gradient is noisier than full Gradient Descent, so your path to the optimum zigzags more. But you take many more steps in the same time, often reaching good solutions much faster.

Why It Matters

SGD makes training on large datasets feasible. Instead of waiting hours for one full-batch update, you get hundreds of noisy updates that collectively move you toward the optimum faster.

Key Takeaway: Stochastic Gradient Descent trades exact gradients for noisy estimates from mini-batches, enabling efficient optimization on large-scale datasets at the cost of a bumpier descent path.