Training vs Evaluation Mode
What you'll learn: How to switch your PyTorch model between training and evaluation modes to control the behavior of dropout and batch normalization layers.
The Core Problem
Certain layers in neural networks behave differently during training versus inference. Specifically:
- Dropout randomly zeros out activations during training to prevent overfitting, but should be turned off during evaluation
- Batch Normalization uses batch statistics during training but switches to running statistics during evaluation
If you forget to switch modes, your model will produce incorrect predictions or unstable results.
The Solution: .train() and .eval()
PyTorch provides two simple methods to control this behavior:
model.train() # Enables training mode
# Dropout is active, BatchNorm uses batch statistics
model.eval() # Enables evaluation mode
# Dropout is disabled, BatchNorm uses running statistics
When to Use Each Mode
During training:
model.train()
for batch in train_loader:
optimizer.zero_grad()
outputs = model(batch)
loss = criterion(outputs, targets)
loss.backward()
optimizer.step()
During evaluation/inference:
model.eval()
with torch.no_grad():
for batch in val_loader:
outputs = model(batch)
# Compute metrics
The Analogy
Think of it like a student taking a practice test versus the real exam. During practice (training), they might use random techniques and experiment. During the real exam (evaluation), they use their proven, consistent knowledge—no randomness allowed.
Key Takeaway: Always call model.train() before training and model.eval() before evaluation to ensure layers like dropout and batch normalization behave correctly for each context.