Course contentsShow
← Machine Learning and Deep Learning
Lesson 756 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Implementing Batch Normalization in PyTorch

Apply nn.BatchNorm1d, BatchNorm2d, and BatchNorm3d with proper placement in network architecture and training/eval modes.

Implementing Batch Normalization in PyTorch

What you'll learn: How to use PyTorch's built-in batch normalization layers and place them correctly in your network architecture.

The Core Idea

PyTorch provides ready-to-use batch normalization layers for different data types: nn.BatchNorm1d for fully connected layers, nn.BatchNorm2d for convolutional layers, and nn.BatchNorm3d for 3D convolutions. You don't need to implement the mathematics yourself—just insert the appropriate layer with the correct number of features.

Using BatchNorm Layers

For fully connected networks:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.BatchNorm1d(256),  # num_features = output of previous layer
    nn.ReLU(),
    nn.Linear(256, 10)
)

For convolutional networks:

conv_model = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3),
    nn.BatchNorm2d(64),  # num_features = number of channels
    nn.ReLU(),
    nn.Conv2d(64, 128, kernel_size=3),
    nn.BatchNorm2d(128)
)

Placement Matters

Batch normalization typically goes after the linear/convolutional layer but before the activation function, though both orderings are used in practice.

Training vs Inference

The model automatically handles train/eval mode:

model.train()  # Uses batch statistics, updates running stats
# ... training loop ...

model.eval()  # Uses running mean/variance, no updates
# ... inference ...

Key Takeaway: Use nn.BatchNorm1d for fully connected layers and nn.BatchNorm2d for CNNs, specifying the number of features/channels; PyTorch handles the train/eval mode switching automatically when you call model.train() or model.eval().