Course contentsShow
← Computer Science
Lesson 2822 of 2,87263. Classical ML AlgorithmsFree lesson

Batch vs Stochastic Gradient Descent

Comparing full-batch updates with single-sample and mini-batch approaches for computational efficiency.

Batch vs Stochastic Gradient Descent

What you'll learn: How different approaches to computing gradients trade off accuracy for speed in training machine learning models.

The Core Idea

When training linear regression (or any model with gradient descent), you need to compute gradients from your data. Batch gradient descent uses all training examples to compute one gradient update, while stochastic gradient descent (SGD) uses just one example at a time. There's also a middle ground: mini-batch gradient descent uses a small subset (e.g., 32 or 64 examples).

Why It Matters

Think of it like getting directions:

  • Batch: Survey everyone in the city before taking a step. Very accurate, but incredibly slow.
  • Stochastic: Ask one random person, take a step, repeat. Fast but noisy—you might zigzag.
  • Mini-batch: Ask a small group, then step. Balanced speed and stability.

Computational Efficiency

Batch gradient descent computes the mean squared error across all samples before updating weights. If you have 1 million examples, that's expensive per update. Updates are smooth and stable, but slow.

Stochastic gradient descent updates weights after each sample. You get 1 million updates in the same time batch takes for one! The learning rate must be carefully tuned because individual samples create noisy gradients that bounce around the minimum.

Mini-batch (typically 32–256 samples) is the practical winner: it vectorizes efficiently on modern hardware (GPUs), reduces gradient noise compared to single samples, and converges much faster than full batch.

Key Takeaway: Batch gradient descent is precise but slow; stochastic is fast but noisy; mini-batch strikes the best balance for most real-world applications by combining computational efficiency with stable convergence.