Checkpointing: Saving Model State
What You'll Learn
You'll learn how to save your model's progress during training so you can resume later or recover from crashes without losing hours of computation.
The Core Idea
Think of checkpointing like saving your progress in a video game. When training deep learning models—which can take hours or even days—you need to periodically save everything necessary to pick up exactly where you left off. This includes not just the model's learned weights, but also the optimizer's internal state and training metadata like which epoch you're on.
What to Save
A complete checkpoint contains three essential components:
- Model weights (
model.state_dict()) - the learned parameters - Optimizer state (
optimizer.state_dict()) - momentum buffers, adaptive learning rates, etc. - Training metadata - current epoch, best validation loss, learning rate schedule state
Implementation
# Saving a checkpoint
checkpoint = {
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'loss': train_loss,
'best_val_loss': best_val_loss
}
torch.save(checkpoint, 'checkpoint.pth')
# Loading a checkpoint
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
start_epoch = checkpoint['epoch'] + 1
best_val_loss = checkpoint['best_val_loss']
Save checkpoints periodically (every few epochs) and always save when you achieve a new best validation performance. This protects against crashes and lets you resume training seamlessly.
Key Takeaway: Use torch.save() with state_dict() to create complete snapshots of your training state, ensuring no progress is ever lost and training can resume exactly where it stopped.