Effective Batch Size vs Physical Batch Size
What you'll learn: How gradient accumulation creates a larger "effective" batch size even when GPU memory limits force you to use smaller physical batches.
The Core Distinction
When training deep learning models, you face a fundamental constraint: GPU memory limits how many samples you can process at once. This physical batch size is what actually fits in memory during a single forward-backward pass.
However, through gradient accumulation, you can simulate training with a much larger batch. The effective batch size is the total number of samples whose gradients contribute to a single optimizer step:
Effective Batch Size = Physical Batch Size × Accumulation Steps
The Restaurant Kitchen Analogy
Think of a restaurant kitchen with limited counter space (GPU memory). You can only prep 4 meals at once (physical batch size = 4), but you want to serve a party of 16 people with identical seasoning across all dishes.
Instead of cooking everything at once, you:
- Prep 4 meals, note the needed adjustments
- Prep 4 more, add to your notes
- Repeat twice more (4 accumulation steps)
- Apply all accumulated adjustments once
You've effectively cooked for 16 people (effective batch size = 16) despite your small counter!
Why This Matters
- Memory efficiency: Train with large effective batches on limited hardware
- Gradient quality: Larger batches provide more stable gradient estimates
- Learning rate scaling: Your effective batch size determines appropriate learning rate (following linear scaling rules from earlier lessons)
- Distributed equivalence: 4 GPUs with batch 8 = 1 GPU with batch 8 and 4 accumulation steps (both give effective batch 32)
Key Takeaway: Physical batch size is constrained by memory; effective batch size is what matters for optimization dynamics, and gradient accumulation lets you decouple the two.