Committing Data Artifacts and Model Outputs
You'll learn best practices for versioning data transformations, features, and model outputs alongside code to maintain reproducible experiments.
The Core Challenge
Your Git repository tracks code beautifully, but data science projects generate much more: transformed datasets, engineered features, trained models, evaluation metrics, and prediction outputs. The question isn't whether to version these artifacts—it's how and what to version effectively.
Smart Versioning Strategies
Version the recipe, not always the cake. Just as a recipe lets you recreate a dish, your code and parameters should let you recreate most artifacts. Commit:
- Lightweight artifacts: Small CSVs of feature importance, confusion matrices, or performance metrics belong in version control
- Model metadata: Training parameters, hyperparameters, dataset versions, and evaluation scores (stored in JSON or YAML)
- Pointers, not files: For large models and datasets, commit references (like file hashes or DVC tracking files) rather than the actual binaries
Organize strategically. Create dedicated directories:
models/metadata/for model cards and config filesresults/for evaluation outputs and figuresfeatures/definitions/for feature engineering documentation
When to Commit vs. External Storage
Commit directly when artifacts are small (<10MB), essential for code review (like baseline model scores), or hard to regenerate (manually annotated data subsets). Use Git LFS or external storage (DVC, cloud buckets) for trained model weights, intermediate datasets, or anything large and easily reproducible.
The golden rule: track enough to reproduce everything, but don't bloat your repository.
Key Takeaway: Version control is not just for code—commit small, critical data artifacts and metadata while using external storage and pointers for large outputs, ensuring both reproducibility and repository efficiency.