Batch Size Effects on Training
What you'll learn: How the number of samples you process at once fundamentally affects your neural network's learning behavior, memory usage, and final performance.
The Core Idea
When training with mini-batch gradient descent, you choose how many samples to process before updating weights. This seemingly simple choice—your batch size—has profound ripple effects throughout training.
Think of batch size like sample size in a survey: a small sample (small batch) gives noisy but frequent feedback, while a large sample (large batch) provides stable but infrequent updates.
Four Key Effects
1. Convergence Speed
Small batches (8-32): Noisy gradients lead to more erratic updates, but you update weights more frequently per epoch. Often reach "good enough" solutions faster in wall-clock time.
Large batches (256-1024+): Smoother, more stable gradient estimates. Each update is more accurate, but you make fewer updates per epoch.
2. Generalization
Small batches introduce gradient noise that acts like a regularizer, helping models escape sharp minima and find flatter regions that generalize better. Large batches can converge to sharper minima with worse test performance (though this is actively researched).
3. Memory Usage
Larger batches require proportionally more GPU memory to store activations for all samples during the forward and backward passes.
4. Gradient Noise
Small batches = high variance gradients (noisier signal)
Large batches = low variance gradients (cleaner but less exploratory)
Practical Trade-offs
Common sweet spot: 32-256 samples, balancing memory efficiency, hardware utilization, and generalization benefits.
Key Takeaway: Batch size is a crucial hyperparameter that trades off gradient noise (good for exploration and generalization) against computational efficiency and stability—smaller batches often generalize better despite noisier training.