Loss and Metric Tracking
What You'll Learn
You'll learn how to correctly accumulate and average losses and metrics across batches, especially when batch sizes vary.
The Core Problem
When training neural networks, you process data in batches. Each batch produces a loss value, but you need to track the overall performance across all batches in an epoch. Simply averaging batch losses can give you incorrect results if batch sizes differ (like when the last batch is smaller).
The Right Way to Track Metrics
Think of grading exams: if one class has 30 students and another has 20, you can't just average the two class averages to get the school average—you need to weight by class size.
For losses and metrics, you must:
- Accumulate the sum: Multiply each batch's loss by its batch size, then add to a running total
- Track total samples: Count how many samples you've processed
- Average at the end: Divide total loss by total samples
# Inside your training loop
total_loss = 0.0
total_samples = 0
for batch_data, batch_labels in dataloader:
outputs = model(batch_data)
loss = criterion(outputs, batch_labels)
# Accumulate weighted loss
batch_size = batch_data.size(0)
total_loss += loss.item() * batch_size
total_samples += batch_size
# Compute epoch average
epoch_loss = total_loss / total_samples
Notice we use loss.item() to extract the Python number (avoiding memory leaks) and weight by batch_size before accumulating.
Why This Matters
If your last batch has 7 samples instead of 32, treating it equally would underweight those 7 examples. Proper weighting ensures every sample contributes fairly to your final metric.
Key Takeaway: Always accumulate loss×batch_size and divide by total samples to get correct epoch averages, especially when batch sizes vary.