Course contentsShow
← Machine Learning and Deep Learning
Lesson 2846 of 3,53861. Experiment Tracking and Model ManagementFree lesson

Data Versioning Best Practices

Strategies for choosing granularity, handling large files, versioning train/test splits, and documenting data changes.

Data Versioning Best Practices

What you'll learn: How to choose the right granularity for versioning data, handle large files efficiently, version train/test splits properly, and document data changes systematically.

Choosing the Right Granularity

Version datasets as logical units, not individual files when possible. If your dataset consists of 10,000 small images, track the entire collection as one version rather than versioning each file separately. This reduces overhead and simplifies rollbacks.

However, for extremely large datasets (multi-TB scale), consider versioning by meaningful partitions—like by date, category, or data source—so you can update portions without re-uploading everything.

Handling Large Files

Always use remote storage (S3, Azure Blob, GCS) with tools like DVC rather than committing large files to Git. Use .dvc files as pointers while the actual data stays in object storage.

Enable deduplication and compression at the storage layer. Content-addressable storage means identical data chunks are stored only once, saving space when datasets evolve incrementally.

Versioning Train/Test Splits

Never version splits separately from the data. Instead, version the split logic itself: commit the random seed, splitting code, or index files that define which samples belong to train/validation/test. This ensures perfect reproducibility—anyone can regenerate identical splits from the versioned raw data.

Store split metadata in your DVC pipeline or as lightweight artifacts alongside model metadata.

Documenting Data Changes

Treat data commits like code commits: write meaningful commit messages explaining what changed and why ("Added 5K labeled samples from Q4 feedback" not "updated data").

Maintain a data changelog or schema registry documenting transformations, feature additions, label corrections, or quality issues. Link data versions to model versions in your registry so you always know which data trained which model.

Key Takeaway: Version datasets as logical units with remote storage, keep splits reproducible through versioned logic rather than static files, and document every data change with context to maintain lineage across experiments.