Setting Random Seeds Correctly
You'll learn how to properly set random seeds across Python, NumPy, PyTorch, and CUDA to ensure reproducible training runs.
The Core Idea
Machine learning training involves many stochastic (random) processes: weight initialization, data shuffling, dropout masks, and more. To reproduce experimental results, you must control all sources of randomness by setting seeds for every library that generates random numbers.
Think of random seeds like planting identical gardens: if you use the same seed packet (number) and growing conditions (code), you'll get the same flowers (results) every time.
Setting Seeds in Practice
You need to set seeds for four distinct systems:
1. Python's Built-in Random Module
import random
random.seed(42)
2. NumPy
import numpy as np
np.random.seed(42)
3. PyTorch CPU Operations
import torch
torch.manual_seed(42)
4. PyTorch CUDA Operations
torch.cuda.manual_seed(42)
torch.cuda.manual_seed_all(42) # For multi-GPU setups
Complete Setup Function
def set_all_seeds(seed=42):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
Call this before creating models, dataloaders, or any random operations.
Important Notes
- Set seeds at the start of your script, before any imports that might trigger random operations
- Each library maintains its own random state—setting one doesn't affect others
- In distributed training, you may want different seeds per process (covered in earlier lessons on distributed training)
Key Takeaway: Reproducible ML experiments require setting random seeds for Python, NumPy, PyTorch CPU, and PyTorch CUDA operations—missing even one source can break reproducibility.