Batch Gradient Descent: Full Dataset Updates
What you'll learn: How batch gradient descent uses every single training example to compute one parameter update, making it the most stable but computationally intensive variant.
The Core Idea
Batch gradient descent is the "classic" form of gradient descent where you calculate the gradient of the cost function using the entire training dataset before taking a single step to update your parameters. Remember the gradient descent update rule from the previous lesson? Batch gradient descent applies it after computing the average gradient across all training examples.
Think of it like getting feedback from every single person in a large auditorium before deciding which direction to walk. It's thorough and gives you the most accurate direction, but you have to wait to hear from everyone before you can move.
How It Works
For each iteration:
- Pass through ALL training examples (the entire batch)
- Compute the cost function using every data point
- Calculate the gradient (average slope across all examples)
- Update parameters once using this complete gradient
- Repeat until convergence
The "batch" in batch gradient descent refers to using the full batch of data. If you have 10,000 training examples, you use all 10,000 to compute one gradient and make one parameter update.
Trade-offs
Advantages: Stable convergence path, smooth cost function reduction, guaranteed to find the minimum for convex problems (like linear regression).
Disadvantages: Computationally expensive for large datasets, slow when you have millions of examples, cannot learn from new data in real-time.
Key Takeaway: Batch gradient descent computes the gradient using all training examples at once, providing the most accurate gradient direction but requiring significant computation per update step.