Implementing Dropout in PyTorch
You'll learn: How to add dropout layers to PyTorch models and correctly handle training versus evaluation modes.
The Practical Implementation
Now that you understand dropout conceptually, it's time to use it in real models. PyTorch provides nn.Dropout as a layer you insert between other layers in your network. The key challenge is understanding that dropout behaves differently during training versus inference.
During training, nn.Dropout(p=0.5) randomly zeros out 50% of inputs and scales the remaining values by 1/(1-p). During evaluation (inference), dropout does nothing—all neurons stay active. PyTorch automatically handles this switch when you call model.train() or model.eval().
Basic Usage Pattern
import torch
import torch.nn as nn
class RegularizedNet(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 256)
self.dropout1 = nn.Dropout(p=0.5)
self.fc2 = nn.Linear(256, 128)
self.dropout2 = nn.Dropout(p=0.3)
self.fc3 = nn.Linear(128, 10)
def forward(self, x):
x = torch.relu(self.fc1(x))
x = self.dropout1(x) # Applied after activation
x = torch.relu(self.fc2(x))
x = self.dropout2(x)
x = self.fc3(x) # No dropout on output layer
return x
# Training mode (dropout active)
model.train()
output = model(train_batch)
# Evaluation mode (dropout disabled)
model.eval()
with torch.no_grad():
predictions = model(test_batch)
Notice dropout is typically placed after activations, not before the final output layer, and each layer can have a different dropout rate.
Key Takeaway: Use nn.Dropout(p) as a layer in your network, place it after activations, and always remember to switch between model.train() and model.eval() modes—forgetting this causes incorrect predictions during inference.