Stochastic Gradient Descent: Single-Sample Updates
What you'll learn: How to update model parameters using just one randomly selected training example at a time instead of the entire dataset.
The Core Idea
In Batch Gradient Descent (lesson 214), you computed the gradient using all training examples before taking a single step. That's accurate but slow, especially with large datasets. Stochastic Gradient Descent (SGD) takes a different approach: pick one random training example, compute the gradient from just that sample, update the parameters immediately, then repeat with another random sample.
Why Single Samples?
Think of learning to throw darts. Batch Gradient Descent is like throwing 1000 darts, carefully measuring where they all landed, calculating the average error, then adjusting your aim once. SGD is like throwing one dart, instantly adjusting based on where that dart landed, throwing another, adjusting again—much faster feedback.
The Trade-off
Each SGD update is noisy—one sample doesn't perfectly represent the entire dataset's gradient. Your path down the cost function landscape becomes zigzagged rather than smooth. But you get to take many more steps in the same amount of time. For a dataset with 10,000 samples, batch gradient descent takes one step per full pass, while SGD takes 10,000 steps.
This noisiness actually has benefits: the random bouncing can help escape shallow local minima and often reaches "good enough" solutions much faster.
When to Use SGD
SGD shines with large datasets where computing the full gradient is expensive. It's also fundamental to training deep neural networks, where even a single full-batch gradient computation might not fit in memory.
Key Takeaway: Stochastic Gradient Descent updates parameters after each single, randomly-selected training example, trading gradient accuracy for speed and enabling much faster iterations through large datasets.