Mini-Batch Gradient Descent
What you'll learn: How mini-batch gradient descent strikes a practical balance between the noise of stochastic updates and the computational cost of full-batch training.
The Goldilocks Approach to Gradient Descent
You've seen two extremes: batch gradient descent computes gradients using the entire dataset (accurate but slow), while stochastic gradient descent (SGD) uses a single sample at a time (fast but noisy). Mini-batch gradient descent is the "just right" middle ground—it computes gradients on small batches of training examples.
How It Works
Instead of using one sample or all samples, you:
- Split your training data into small batches (typically 32, 64, 128, or 256 samples)
- Compute the gradient by averaging the loss over that mini-batch
- Update weights based on this mini-batch gradient
- Move to the next mini-batch and repeat
Think of it like taste-testing soup: you don't need to consume the entire pot (full batch) or just one molecule (single sample) to judge the flavor—a spoonful (mini-batch) gives you a reliable sense while being practical.
Why Mini-Batches Win
Noise reduction: Averaging gradients across multiple samples smooths out the extreme randomness of single-sample updates, leading to more stable convergence.
Computational efficiency: Modern hardware (GPUs) excels at parallel matrix operations. Processing 64 samples together is often nearly as fast as processing one, thanks to vectorization.
Memory feasibility: Full-batch gradient descent becomes impossible with large datasets that don't fit in memory. Mini-batches keep memory usage bounded.
Generalization: The slight noise in mini-batch gradients actually acts as a regularizer, helping models escape sharp minima and find flatter, more generalizable solutions.
Key Takeaway: Mini-batch gradient descent balances computational efficiency with gradient stability by computing updates on small batches, making it the standard choice for training deep neural networks in practice.