Managing Python Dependencies with requirements.txt
You'll learn how to create and maintain requirements.txt files with pinned versions to guarantee identical package installations across environments and team members.
Why Dependency Management Matters for Reproducibility
Imagine training a model today, then trying to reproduce your results six months later—only to find that a library update broke your code or changed numerical behavior. Without explicit dependency tracking, you're building on shifting sand.
In ML, where tiny numerical differences can compound through training, locking down exact package versions is critical for reproducibility. This is where requirements.txt comes in: a simple text file that lists every Python package your project needs, with specific version numbers.
Creating a requirements.txt File
The basic format is straightforward:
torch==2.0.1
numpy==1.24.3
transformers==4.30.2
Use == to pin exact versions. This ensures that anyone running pip install -r requirements.txt gets precisely the same packages you used.
Generate automatically from your current environment:
pip freeze > requirements.txt
This captures all installed packages with their versions—though it may include unnecessary dependencies.
Best Practices for ML Projects
- Pin major packages explicitly: Always specify exact versions for core ML libraries (PyTorch, TensorFlow, transformers)
- Include indirect dependencies: Critical packages like
numpyorpillowshould be pinned too - Update mindfully: When upgrading, test thoroughly and document why in commit messages
- Separate dev dependencies: Consider
requirements-dev.txtfor testing/linting tools - Version control it: Commit
requirements.txtalongside your code and model checkpoints
Think of requirements.txt as a snapshot of your software environment—essential metadata for reproducing experiments months or years later.
Key Takeaway: Pinning exact package versions in requirements.txt eliminates a major source of irreproducibility in ML, ensuring your experiments remain runnable and your results verifiable over time.