Mini-Batch Gradient Descent: The Practical Middle Ground
What you'll learn: How mini-batch gradient descent combines the stability of batch gradient descent with the speed of stochastic gradient descent by using small random subsets of data.
The Goldilocks Approach
You've seen two extremes: batch gradient descent uses all data points at once (accurate but slow), while stochastic gradient descent uses one point at a time (fast but noisy). Mini-batch gradient descent is the "just right" compromise.
How It Works
Instead of computing gradients on the entire dataset or a single sample, mini-batch gradient descent:
- Randomly divides your training data into small groups (batches)
- Computes the gradient using only the samples in one mini-batch
- Updates the parameters based on that mini-batch's gradient
- Moves to the next batch and repeats
Common mini-batch sizes are 32, 64, 128, or 256 samples.
Why This Works Best
Computational efficiency: You update parameters more frequently than batch gradient descent, making progress faster through the cost function landscape.
Better gradient estimates: Averaging over multiple samples (unlike SGD's single sample) gives a more stable direction to move in, reducing the update noise.
Hardware optimization: Modern GPUs are designed to process batches of data efficiently, making mini-batch sizes like 32 or 64 run much faster than processing samples one-by-one.
Memory management: You don't need to hold the entire dataset in memory at once, unlike full-batch gradient descent.
Think of it like getting restaurant feedback: surveying all customers takes forever (batch), asking one person is unreliable (stochastic), but asking a small group of 50 gives you actionable insights quickly (mini-batch).
Key Takeaway: Mini-batch gradient descent balances convergence stability and computational speed by computing gradients over small random subsets, making it the standard choice in modern machine learning.