Course contentsShow
← Machine Learning and Deep Learning
Lesson 2854 of 3,53861. Experiment Tracking and Model ManagementFree lesson

Environment Management with Poetry and Pipenv

Use modern dependency managers that handle lock files and transitive dependencies automatically.

Environment Management with Poetry and Pipenv

You'll learn: How modern dependency managers automatically handle lock files and transitive dependencies to ensure reproducible ML environments.

Why Modern Dependency Managers Matter

While requirements.txt and virtual environments (from lesson 2851-2852) give you basic isolation, they don't solve a critical problem: transitive dependencies. When you install transformers, it pulls in dozens of other packages, each with their own version requirements. Traditional pip doesn't lock these sub-dependencies, so running pip install -r requirements.txt on different days can produce different environments.

Poetry and Pipenv: The Lock File Solution

Both Poetry and Pipenv introduce a two-file system:

  1. Dependency specification file (pyproject.toml for Poetry, Pipfile for Pipenv) — Lists what you directly need
  2. Lock file (poetry.lock or Pipfile.lock) — Records exact versions of every package, including all transitive dependencies

Think of it like a restaurant recipe (your direct dependencies) versus the complete ingredient manifest with supplier batch numbers (the lock file). The lock file ensures anyone can recreate the exact environment.

How It Works

When you add torch==2.0.0, Poetry/Pipenv:

  • Resolves all dependencies torch needs
  • Checks for version conflicts across your entire dependency tree
  • Writes exact versions (including checksums) to the lock file
  • Future installs use the lock file, guaranteeing identical environments

This is crucial for ML reproducibility: the same model code with slightly different package versions can produce different results due to numerical implementation differences.

Poetry vs Pipenv Choice

Poetry has become more popular in the ML community for its faster resolver and cleaner pyproject.toml format. Pipenv integrates tightly with pip conventions. Both dramatically improve upon requirements.txt for production ML projects.

Key Takeaway: Poetry and Pipenv use lock files to pin every transitive dependency, ensuring your ML environment is perfectly reproducible across machines and time—essential for experiment reproducibility and deployment consistency.