Course contentsShow
← Machine Learning and Deep Learning
Lesson 831 of 3,53819. PyTorch FundamentalsFree lesson

Loss and Metric Tracking

Accumulating and averaging losses and metrics across batches, handling batch size variations correctly.

Loss and Metric Tracking

What You'll Learn

You'll learn how to correctly accumulate and average losses and metrics across batches, especially when batch sizes vary.

The Core Problem

When training neural networks, you process data in batches. Each batch produces a loss value, but you need to track the overall performance across all batches in an epoch. Simply averaging batch losses can give you incorrect results if batch sizes differ (like when the last batch is smaller).

The Right Way to Track Metrics

Think of grading exams: if one class has 30 students and another has 20, you can't just average the two class averages to get the school average—you need to weight by class size.

For losses and metrics, you must:

  1. Accumulate the sum: Multiply each batch's loss by its batch size, then add to a running total
  2. Track total samples: Count how many samples you've processed
  3. Average at the end: Divide total loss by total samples
# Inside your training loop
total_loss = 0.0
total_samples = 0

for batch_data, batch_labels in dataloader:
    outputs = model(batch_data)
    loss = criterion(outputs, batch_labels)
    
    # Accumulate weighted loss
    batch_size = batch_data.size(0)
    total_loss += loss.item() * batch_size
    total_samples += batch_size

# Compute epoch average
epoch_loss = total_loss / total_samples

Notice we use loss.item() to extract the Python number (avoiding memory leaks) and weight by batch_size before accumulating.

Why This Matters

If your last batch has 7 samples instead of 32, treating it equally would underweight those 7 examples. Proper weighting ensures every sample contributes fairly to your final metric.

Key Takeaway: Always accumulate loss×batch_size and divide by total samples to get correct epoch averages, especially when batch sizes vary.