Batch Normalization: Train vs Inference Mode
What you'll learn: How batch normalization behaves differently during training versus inference, and why this distinction is crucial for correct model deployment.
The Critical Difference
Batch normalization operates in two fundamentally different modes:
Training mode: Uses statistics computed from the current mini-batch. For each batch, BN calculates the mean and variance from those specific samples, then normalizes using those values. This introduces helpful noise and regularization.
Inference mode: Uses running estimates of the population mean and variance accumulated during training. At test time, you typically process one sample at a time—there is no "batch" to compute statistics from.
Why This Matters
During training, your network learns to normalize using batch-specific statistics. But at inference, you need consistent, stable normalization that doesn't depend on which other samples happen to be in the batch.
The solution: While training, batch normalization maintains running averages (exponential moving averages) of the mean and variance it sees across all batches. These running statistics approximate the true population parameters.
The Switching Mechanism
Think of it like a restaurant:
- Training mode: The chef tastes and adjusts each dish based on the ingredients in that particular batch
- Inference mode: The chef follows the perfected recipe developed from experience with thousands of previous batches
When you call model.train(), BN layers compute batch statistics and update running averages. When you call model.eval(), BN layers freeze and use only the stored running statistics.
Key Takeaway: Batch normalization must switch from batch statistics (training) to running population estimates (inference) to ensure consistent, reliable predictions on individual samples or small batches at test time.