Course contentsShow
← Machine Learning and Deep Learning
Lesson 827 of 3,53819. PyTorch FundamentalsFree lesson

Anatomy of a Training Loop

Understanding the standard training loop structure: forward pass, loss computation, backward pass, and optimizer step.

Anatomy of a Training Loop

What You'll Learn

You'll learn the four essential steps that happen in every PyTorch training iteration: forward pass, loss computation, backward pass, and optimizer step.

The Four Core Steps

Think of training a neural network like teaching someone to throw darts: they throw (forward pass), you measure how far they missed (loss), they learn what went wrong (backward pass), and they adjust their technique (optimizer step). This cycle repeats until they improve.

1. Forward Pass

Your model processes input data and produces predictions. You call model(inputs) to get outputs by passing data through all the layers.

2. Loss Computation

You compare predictions to the true labels using a loss function. This quantifies how wrong your model is: loss = loss_fn(outputs, targets).

3. Backward Pass

This is where autograd shines. Calling loss.backward() computes gradients for all parameters with requires_grad=True, tracing back through the computational graph.

4. Optimizer Step

The optimizer uses those gradients to update model weights: optimizer.step(). This adjusts parameters to reduce the loss.

The Critical Detail: Zero Gradients

Before each iteration, you must call optimizer.zero_grad() to clear old gradients. PyTorch accumulates gradients by default, so forgetting this means you're adding new gradients to stale ones from previous batches.

The Pattern

for inputs, targets in dataloader:
    optimizer.zero_grad()        # Clear old gradients
    outputs = model(inputs)      # Forward pass
    loss = loss_fn(outputs, targets)  # Compute loss
    loss.backward()              # Backward pass
    optimizer.step()             # Update weights

Key Takeaway: Every training loop follows the same four-step rhythm: forward, loss, backward, optimize—with gradient zeroing before each iteration to prevent accumulation bugs.