Course contentsShow
← Machine Learning and Deep Learning
Lesson 673 of 3,53816. Activation Functions and Weight InitializationFree lesson

Implementing Initialization in PyTorch

Using torch.nn.init methods to apply Xavier, He, and custom initialization schemes to layers.

Implementing Initialization in PyTorch

What you'll learn: How to apply Xavier, He, and custom initialization schemes to PyTorch layers using torch.nn.init methods.

The Practical Side of Initialization

You've learned why Xavier and He initialization matter and how they preserve variance. Now it's time to actually use them. PyTorch provides the torch.nn.init module with ready-to-use functions that apply these initialization schemes to your network's weights and biases.

Using torch.nn.init Methods

Think of torch.nn.init as a toolbox of initialization strategies. Instead of manually setting random values, you call specialized functions that follow mathematical principles like Xavier or He initialization.

Common Initialization Functions

Xavier (Glorot) Initialization:

import torch.nn as nn
import torch.nn.init as init

layer = nn.Linear(100, 50)
init.xavier_uniform_(layer.weight)  # Uniform distribution
init.xavier_normal_(layer.weight)   # Normal distribution

He Initialization:

init.kaiming_uniform_(layer.weight, nonlinearity='relu')
init.kaiming_normal_(layer.weight, nonlinearity='relu')

Custom Initialization:

init.constant_(layer.bias, 0.0)           # Set all biases to 0
init.uniform_(layer.weight, -0.1, 0.1)    # Uniform in range
init.normal_(layer.weight, mean=0, std=0.01)  # Custom normal

Applying to Entire Networks

def initialize_weights(model):
    for module in model.modules():
        if isinstance(module, nn.Linear):
            init.kaiming_normal_(module.weight, nonlinearity='relu')
            init.constant_(module.bias, 0)

Notice the underscore suffix (_) — this indicates in-place operations that modify tensors directly.

Key Takeaway: PyTorch's torch.nn.init module provides ready-made initialization functions like xavier_normal_, kaiming_uniform_, and custom options, letting you apply proper initialization schemes to any layer with a single function call.