Anatomy of a Training Loop
What You'll Learn
You'll learn the four essential steps that happen in every PyTorch training iteration: forward pass, loss computation, backward pass, and optimizer step.
The Four Core Steps
Think of training a neural network like teaching someone to throw darts: they throw (forward pass), you measure how far they missed (loss), they learn what went wrong (backward pass), and they adjust their technique (optimizer step). This cycle repeats until they improve.
1. Forward Pass
Your model processes input data and produces predictions. You call model(inputs) to get outputs by passing data through all the layers.
2. Loss Computation
You compare predictions to the true labels using a loss function. This quantifies how wrong your model is: loss = loss_fn(outputs, targets).
3. Backward Pass
This is where autograd shines. Calling loss.backward() computes gradients for all parameters with requires_grad=True, tracing back through the computational graph.
4. Optimizer Step
The optimizer uses those gradients to update model weights: optimizer.step(). This adjusts parameters to reduce the loss.
The Critical Detail: Zero Gradients
Before each iteration, you must call optimizer.zero_grad() to clear old gradients. PyTorch accumulates gradients by default, so forgetting this means you're adding new gradients to stale ones from previous batches.
The Pattern
for inputs, targets in dataloader:
optimizer.zero_grad() # Clear old gradients
outputs = model(inputs) # Forward pass
loss = loss_fn(outputs, targets) # Compute loss
loss.backward() # Backward pass
optimizer.step() # Update weights
Key Takeaway: Every training loop follows the same four-step rhythm: forward, loss, backward, optimize—with gradient zeroing before each iteration to prevent accumulation bugs.