Course contentsShow
← Data Science
Lesson 2070 of 2,14547. Reproducible ResearchFree lesson

Separating Data from Code

Understand why raw data should never live in version control and how to reference it from code.

Separating Data from Code

You'll learn why keeping raw data out of version control protects your project and how to reference external data properly.

Why Data Doesn't Belong in Git

Imagine storing your entire music library inside every email you send about a playlist—absurd, right? The same principle applies to version control. Git tracks changes to text efficiently, but raw data files are typically:

  • Large: CSVs, images, and databases bloat your repository, making clones slow
  • Binary or semi-structured: Git can't show meaningful diffs, so every change duplicates the entire file
  • Subject to different update cycles: Data changes frequently and independently from your analysis code

The Better Approach: Reference, Don't Include

Your code should reference where data lives, not contain it. Think of your script as a recipe that says "get eggs from the fridge" rather than embedding actual eggs in the recipe card.

Practical implementation:

  1. Store data separately: Use cloud storage (S3, Google Cloud), shared drives, or dedicated data warehouses
  2. Use configuration files: Create a config.yml or .env file (gitignored!) with data paths
  3. Document data sources: In your README, specify where data lives and how to access it
  4. Use relative paths: data/raw/sales.csv assumes data exists locally in a standard structure
# Good: Reference external data
import pandas as pd

data_path = os.getenv('DATA_PATH')  # From environment variable
df = pd.read_csv(f"{data_path}/raw/sales.csv")

For small sample/test datasets needed for CI/CD, consider Git LFS or storing truly minimal examples (<1MB) directly.

Key Takeaway: Keep raw data out of version control by storing it externally and referencing it through configuration—this keeps repositories lean, clones fast, and separates data governance from code evolution.