Stochastic Gradient Descent Basics
What you'll learn: How to speed up optimization on massive datasets by using random mini-batches instead of the entire dataset at each step.
The Core Problem
You've learned that Gradient Descent computes the gradient using all your training data at every iteration. Imagine you have 1 million examples—computing the full gradient means processing all 1 million points just to take one step! This becomes painfully slow.
The Stochastic Solution
Stochastic Gradient Descent (SGD) takes a smarter approach: instead of computing the exact gradient from all data, it estimates the gradient using a small random subset called a mini-batch (often 32, 64, or 256 examples).
Think of it like a political poll. You don't need to ask all 300 million Americans their opinion—a random sample of 1,000 people gives you a noisy but useful estimate of the population's view. Similarly, a mini-batch gives you a noisy gradient estimate that on average points in the right direction.
How It Works
At each iteration:
- Randomly select a mini-batch from your dataset
- Compute the gradient using only those examples
- Take a step in that (approximate) direction
- Repeat with a new random mini-batch
The gradient is noisier than full Gradient Descent, so your path to the optimum zigzags more. But you take many more steps in the same time, often reaching good solutions much faster.
Why It Matters
SGD makes training on large datasets feasible. Instead of waiting hours for one full-batch update, you get hundreds of noisy updates that collectively move you toward the optimum faster.
Key Takeaway: Stochastic Gradient Descent trades exact gradients for noisy estimates from mini-batches, enabling efficient optimization on large-scale datasets at the cost of a bumpier descent path.