Course contentsShow
← Machine Learning and Deep Learning
Lesson 2783 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Effective Batch Size vs Physical Batch Size

Distinguishing between the physical batch size that fits in memory and the effective batch size achieved through accumulation steps.

Effective Batch Size vs Physical Batch Size

What you'll learn: How gradient accumulation creates a larger "effective" batch size even when GPU memory limits force you to use smaller physical batches.

The Core Distinction

When training deep learning models, you face a fundamental constraint: GPU memory limits how many samples you can process at once. This physical batch size is what actually fits in memory during a single forward-backward pass.

However, through gradient accumulation, you can simulate training with a much larger batch. The effective batch size is the total number of samples whose gradients contribute to a single optimizer step:

Effective Batch Size = Physical Batch Size × Accumulation Steps

The Restaurant Kitchen Analogy

Think of a restaurant kitchen with limited counter space (GPU memory). You can only prep 4 meals at once (physical batch size = 4), but you want to serve a party of 16 people with identical seasoning across all dishes.

Instead of cooking everything at once, you:

  1. Prep 4 meals, note the needed adjustments
  2. Prep 4 more, add to your notes
  3. Repeat twice more (4 accumulation steps)
  4. Apply all accumulated adjustments once

You've effectively cooked for 16 people (effective batch size = 16) despite your small counter!

Why This Matters

  • Memory efficiency: Train with large effective batches on limited hardware
  • Gradient quality: Larger batches provide more stable gradient estimates
  • Learning rate scaling: Your effective batch size determines appropriate learning rate (following linear scaling rules from earlier lessons)
  • Distributed equivalence: 4 GPUs with batch 8 = 1 GPU with batch 8 and 4 accumulation steps (both give effective batch 32)

Key Takeaway: Physical batch size is constrained by memory; effective batch size is what matters for optimization dynamics, and gradient accumulation lets you decouple the two.