Implementing Initialization in PyTorch
What you'll learn: How to apply Xavier, He, and custom initialization schemes to PyTorch layers using torch.nn.init methods.
The Practical Side of Initialization
You've learned why Xavier and He initialization matter and how they preserve variance. Now it's time to actually use them. PyTorch provides the torch.nn.init module with ready-to-use functions that apply these initialization schemes to your network's weights and biases.
Using torch.nn.init Methods
Think of torch.nn.init as a toolbox of initialization strategies. Instead of manually setting random values, you call specialized functions that follow mathematical principles like Xavier or He initialization.
Common Initialization Functions
Xavier (Glorot) Initialization:
import torch.nn as nn
import torch.nn.init as init
layer = nn.Linear(100, 50)
init.xavier_uniform_(layer.weight) # Uniform distribution
init.xavier_normal_(layer.weight) # Normal distribution
He Initialization:
init.kaiming_uniform_(layer.weight, nonlinearity='relu')
init.kaiming_normal_(layer.weight, nonlinearity='relu')
Custom Initialization:
init.constant_(layer.bias, 0.0) # Set all biases to 0
init.uniform_(layer.weight, -0.1, 0.1) # Uniform in range
init.normal_(layer.weight, mean=0, std=0.01) # Custom normal
Applying to Entire Networks
def initialize_weights(model):
for module in model.modules():
if isinstance(module, nn.Linear):
init.kaiming_normal_(module.weight, nonlinearity='relu')
init.constant_(module.bias, 0)
Notice the underscore suffix (_) — this indicates in-place operations that modify tensors directly.
Key Takeaway: PyTorch's torch.nn.init module provides ready-made initialization functions like xavier_normal_, kaiming_uniform_, and custom options, letting you apply proper initialization schemes to any layer with a single function call.