Implementing Batch Normalization in PyTorch
What you'll learn: How to use PyTorch's built-in batch normalization layers and place them correctly in your network architecture.
The Core Idea
PyTorch provides ready-to-use batch normalization layers for different data types: nn.BatchNorm1d for fully connected layers, nn.BatchNorm2d for convolutional layers, and nn.BatchNorm3d for 3D convolutions. You don't need to implement the mathematics yourself—just insert the appropriate layer with the correct number of features.
Using BatchNorm Layers
For fully connected networks:
import torch.nn as nn
model = nn.Sequential(
nn.Linear(784, 256),
nn.BatchNorm1d(256), # num_features = output of previous layer
nn.ReLU(),
nn.Linear(256, 10)
)
For convolutional networks:
conv_model = nn.Sequential(
nn.Conv2d(3, 64, kernel_size=3),
nn.BatchNorm2d(64), # num_features = number of channels
nn.ReLU(),
nn.Conv2d(64, 128, kernel_size=3),
nn.BatchNorm2d(128)
)
Placement Matters
Batch normalization typically goes after the linear/convolutional layer but before the activation function, though both orderings are used in practice.
Training vs Inference
The model automatically handles train/eval mode:
model.train() # Uses batch statistics, updates running stats
# ... training loop ...
model.eval() # Uses running mean/variance, no updates
# ... inference ...
Key Takeaway: Use nn.BatchNorm1d for fully connected layers and nn.BatchNorm2d for CNNs, specifying the number of features/channels; PyTorch handles the train/eval mode switching automatically when you call model.train() or model.eval().