Course contentsShow
← Machine Learning and Deep Learning
Lesson 749 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Implementing Dropout in PyTorch

Using nn.Dropout layers correctly and understanding training vs evaluation mode behavior.

Implementing Dropout in PyTorch

You'll learn: How to add dropout layers to PyTorch models and correctly handle training versus evaluation modes.

The Practical Implementation

Now that you understand dropout conceptually, it's time to use it in real models. PyTorch provides nn.Dropout as a layer you insert between other layers in your network. The key challenge is understanding that dropout behaves differently during training versus inference.

During training, nn.Dropout(p=0.5) randomly zeros out 50% of inputs and scales the remaining values by 1/(1-p). During evaluation (inference), dropout does nothing—all neurons stay active. PyTorch automatically handles this switch when you call model.train() or model.eval().

Basic Usage Pattern

import torch
import torch.nn as nn

class RegularizedNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.dropout1 = nn.Dropout(p=0.5)
        self.fc2 = nn.Linear(256, 128)
        self.dropout2 = nn.Dropout(p=0.3)
        self.fc3 = nn.Linear(128, 10)
    
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout1(x)  # Applied after activation
        x = torch.relu(self.fc2(x))
        x = self.dropout2(x)
        x = self.fc3(x)  # No dropout on output layer
        return x

# Training mode (dropout active)
model.train()
output = model(train_batch)

# Evaluation mode (dropout disabled)
model.eval()
with torch.no_grad():
    predictions = model(test_batch)

Notice dropout is typically placed after activations, not before the final output layer, and each layer can have a different dropout rate.

Key Takeaway: Use nn.Dropout(p) as a layer in your network, place it after activations, and always remember to switch between model.train() and model.eval() modes—forgetting this causes incorrect predictions during inference.