Data Versioning vs Code Versioning
What you'll learn: How versioning data differs fundamentally from versioning code, and why you can't just use Git for your datasets.
The Core Difference
Code versioning (like Git) and data versioning solve similar problems—tracking changes over time—but they face completely different constraints. Code files are small, text-based, and change frequently through line-by-line edits. Data in ML, however, consists of large binary files (images, videos, model weights) that are immutable: you don't edit a JPEG line-by-line; you replace it entirely.
Why Git Doesn't Work for Data
Size and Storage
Code repositories typically measure in megabytes. ML datasets measure in gigabytes or terabytes. Git's diff algorithm stores every version's changes—perfect for text, catastrophic for a 50GB video dataset where even tiny changes create massive diffs.
Binary File Challenge
Git compresses text beautifully but struggles with binary files. Changing one pixel in a million images creates a million "new" files from Git's perspective, bloating your repository.
Immutability vs Mutability
Code evolves incrementally (edit function, fix bug). Data snapshots are immutable: "raw_data_v1.parquet" doesn't get edited—you create "raw_data_v2.parquet" after preprocessing. This requires content-addressed storage (hash-based) rather than delta-based storage.
The Solution Space
Data versioning tools (DVC, LakeFS) use pointers: your Git repo stores lightweight metadata files (~KB) that reference actual data stored externally (S3, NFS). Think of it like Git storing a library card catalog instead of all the books.
Key Takeaway: Data versioning requires specialized tools because ML data is large, binary, and immutable—fundamentally incompatible with Git's line-diff model designed for small, text-based, incrementally-edited code files.