Course contentsShow
← Machine Learning and Deep Learning
Lesson 828 of 3,53819. PyTorch FundamentalsFree lesson

Training vs Evaluation Mode

Using model.train() and model.eval() to control batch normalization and dropout behavior during training and inference.

Training vs Evaluation Mode

What you'll learn: How to switch your PyTorch model between training and evaluation modes to control the behavior of dropout and batch normalization layers.

The Core Problem

Certain layers in neural networks behave differently during training versus inference. Specifically:

  • Dropout randomly zeros out activations during training to prevent overfitting, but should be turned off during evaluation
  • Batch Normalization uses batch statistics during training but switches to running statistics during evaluation

If you forget to switch modes, your model will produce incorrect predictions or unstable results.

The Solution: .train() and .eval()

PyTorch provides two simple methods to control this behavior:

model.train()  # Enables training mode
# Dropout is active, BatchNorm uses batch statistics

model.eval()   # Enables evaluation mode
# Dropout is disabled, BatchNorm uses running statistics

When to Use Each Mode

During training:

model.train()
for batch in train_loader:
    optimizer.zero_grad()
    outputs = model(batch)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()

During evaluation/inference:

model.eval()
with torch.no_grad():
    for batch in val_loader:
        outputs = model(batch)
        # Compute metrics

The Analogy

Think of it like a student taking a practice test versus the real exam. During practice (training), they might use random techniques and experiment. During the real exam (evaluation), they use their proven, consistent knowledge—no randomness allowed.

Key Takeaway: Always call model.train() before training and model.eval() before evaluation to ensure layers like dropout and batch normalization behave correctly for each context.