Separating Data from Code
You'll learn why keeping raw data out of version control protects your project and how to reference external data properly.
Why Data Doesn't Belong in Git
Imagine storing your entire music library inside every email you send about a playlist—absurd, right? The same principle applies to version control. Git tracks changes to text efficiently, but raw data files are typically:
- Large: CSVs, images, and databases bloat your repository, making clones slow
- Binary or semi-structured: Git can't show meaningful diffs, so every change duplicates the entire file
- Subject to different update cycles: Data changes frequently and independently from your analysis code
The Better Approach: Reference, Don't Include
Your code should reference where data lives, not contain it. Think of your script as a recipe that says "get eggs from the fridge" rather than embedding actual eggs in the recipe card.
Practical implementation:
- Store data separately: Use cloud storage (S3, Google Cloud), shared drives, or dedicated data warehouses
- Use configuration files: Create a
config.ymlor.envfile (gitignored!) with data paths - Document data sources: In your README, specify where data lives and how to access it
- Use relative paths:
data/raw/sales.csvassumes data exists locally in a standard structure
# Good: Reference external data
import pandas as pd
data_path = os.getenv('DATA_PATH') # From environment variable
df = pd.read_csv(f"{data_path}/raw/sales.csv")
For small sample/test datasets needed for CI/CD, consider Git LFS or storing truly minimal examples (<1MB) directly.
Key Takeaway: Keep raw data out of version control by storing it externally and referencing it through configuration—this keeps repositories lean, clones fast, and separates data governance from code evolution.