Sources of Non-Determinism in ML
What You'll Learn
You'll learn to identify the primary culprits that make machine learning experiments hard to reproduce, even with identical code and data.
The Reproducibility Challenge
Imagine running the same ML training script twice and getting different results each time. Frustrating, right? Non-determinism in ML means your experiments can produce varying outcomes despite seemingly identical conditions. Understanding where randomness sneaks in is critical for reproducible research and reliable model development.
Common Sources of Non-Determinism
Weight Initialization
Neural networks start with random weights. Even with the same random seed, different library versions or hardware may produce different initial values, causing training trajectories to diverge.
Data Shuffling
Most training loops shuffle data each epoch. Without careful seeding, the order varies between runs, affecting gradient updates and final model states—especially with small batch sizes or limited data.
Hardware Differences
GPUs use non-deterministic algorithms for performance (like atomic operations in reductions). The same code on different GPU models—or even the same model—can yield different floating-point arithmetic results due to operation scheduling and parallelism.
Library Versions
Updates to PyTorch, TensorFlow, CUDA, or cuDNN can change underlying implementations. A function that was deterministic in version 1.10 might use a faster, non-deterministic algorithm in 1.11.
Other Sources
Dropout layers, data augmentation, multi-threaded data loading, and distributed training all introduce randomness that compounds across training.
Think of It Like Cooking
You're following a recipe (your code), but the oven temperature fluctuates (hardware), your measuring cups are slightly different (library versions), and you stir in random directions (data shuffling). Small variations cascade into noticeably different outcomes.
Key Takeaway: Non-determinism in ML stems from random weight initialization, data shuffling, hardware-level operations, and library version differences—all of which must be controlled for true reproducibility.