Course contentsShow
← Machine Learning and Deep Learning
Lesson 684 of 3,53817. Optimization for Deep LearningFree lesson

Mini-Batch Gradient Descent

Balancing noise and efficiency by computing gradients on small batches rather than single samples or full datasets.

Mini-Batch Gradient Descent

What you'll learn: How mini-batch gradient descent strikes a practical balance between the noise of stochastic updates and the computational cost of full-batch training.

The Goldilocks Approach to Gradient Descent

You've seen two extremes: batch gradient descent computes gradients using the entire dataset (accurate but slow), while stochastic gradient descent (SGD) uses a single sample at a time (fast but noisy). Mini-batch gradient descent is the "just right" middle ground—it computes gradients on small batches of training examples.

How It Works

Instead of using one sample or all samples, you:

  1. Split your training data into small batches (typically 32, 64, 128, or 256 samples)
  2. Compute the gradient by averaging the loss over that mini-batch
  3. Update weights based on this mini-batch gradient
  4. Move to the next mini-batch and repeat

Think of it like taste-testing soup: you don't need to consume the entire pot (full batch) or just one molecule (single sample) to judge the flavor—a spoonful (mini-batch) gives you a reliable sense while being practical.

Why Mini-Batches Win

Noise reduction: Averaging gradients across multiple samples smooths out the extreme randomness of single-sample updates, leading to more stable convergence.

Computational efficiency: Modern hardware (GPUs) excels at parallel matrix operations. Processing 64 samples together is often nearly as fast as processing one, thanks to vectorization.

Memory feasibility: Full-batch gradient descent becomes impossible with large datasets that don't fit in memory. Mini-batches keep memory usage bounded.

Generalization: The slight noise in mini-batch gradients actually acts as a regularizer, helping models escape sharp minima and find flatter, more generalizable solutions.

Key Takeaway: Mini-batch gradient descent balances computational efficiency with gradient stability by computing updates on small batches, making it the standard choice for training deep neural networks in practice.