ModuleList and ModuleDict for Dynamic Architectures
What You'll Learn
You'll learn how to use nn.ModuleList and nn.ModuleDict to build models with variable numbers of layers or named layer collections that PyTorch can properly track.
Why We Need Them
When you create a custom nn.Module, PyTorch automatically registers submodules you assign as attributes. But what if you don't know how many layers you need in advance, or want to store them in a collection? Using a regular Python list or dict won't work—PyTorch won't register those layers' parameters!
Think of it like this: PyTorch needs to know about all your model's layers to track their parameters for training. Regular Python containers are "invisible" to PyTorch, but ModuleList and ModuleDict are "visible."
Using ModuleList
Use nn.ModuleList when you need a variable number of similar layers:
import torch.nn as nn
class DynamicNet(nn.Module):
def __init__(self, num_layers, hidden_size):
super().__init__()
self.layers = nn.ModuleList([
nn.Linear(hidden_size, hidden_size)
for _ in range(num_layers)
])
def forward(self, x):
for layer in self.layers:
x = layer(x)
return x
Using ModuleDict
Use nn.ModuleDict when you need named access to different layer options:
class MultiTaskNet(nn.Module):
def __init__(self):
super().__init__()
self.shared = nn.Linear(128, 64)
self.heads = nn.ModuleDict({
'classification': nn.Linear(64, 10),
'regression': nn.Linear(64, 1)
})
def forward(self, x, task):
x = self.shared(x)
return self.heads[task](x)
Key Takeaway: Use nn.ModuleList for variable-length layer sequences and nn.ModuleDict for named layer collections—both ensure PyTorch properly registers all parameters for training.