Batch vs Stochastic Gradient Descent
What you'll learn: How different approaches to computing gradients trade off accuracy for speed in training machine learning models.
The Core Idea
When training linear regression (or any model with gradient descent), you need to compute gradients from your data. Batch gradient descent uses all training examples to compute one gradient update, while stochastic gradient descent (SGD) uses just one example at a time. There's also a middle ground: mini-batch gradient descent uses a small subset (e.g., 32 or 64 examples).
Why It Matters
Think of it like getting directions:
- Batch: Survey everyone in the city before taking a step. Very accurate, but incredibly slow.
- Stochastic: Ask one random person, take a step, repeat. Fast but noisy—you might zigzag.
- Mini-batch: Ask a small group, then step. Balanced speed and stability.
Computational Efficiency
Batch gradient descent computes the mean squared error across all samples before updating weights. If you have 1 million examples, that's expensive per update. Updates are smooth and stable, but slow.
Stochastic gradient descent updates weights after each sample. You get 1 million updates in the same time batch takes for one! The learning rate must be carefully tuned because individual samples create noisy gradients that bounce around the minimum.
Mini-batch (typically 32–256 samples) is the practical winner: it vectorizes efficiently on modern hardware (GPUs), reduces gradient noise compared to single samples, and converges much faster than full batch.
Key Takeaway: Batch gradient descent is precise but slow; stochastic is fast but noisy; mini-batch strikes the best balance for most real-world applications by combining computational efficiency with stable convergence.