Git Large File Storage (LFS) for Data Assets
You'll learn when and why to use Git LFS to handle large datasets and model files that would otherwise overwhelm standard Git repositories.
The Problem with Large Files in Git
Git tracks every version of every file in full. When you commit a 500 MB dataset, then modify it, Git stores both versions completely—your repository balloons to 1 GB. Do this ten times and your repo becomes unwieldy, slow to clone, and painful to work with. Git wasn't designed for large binary files like datasets, trained models, or database dumps.
How Git LFS Solves This
Git LFS (Large File Storage) replaces large files in your repository with small pointer files. The actual large files live on a separate LFS server (like GitHub LFS). When you clone the repo, you get the pointers by default; the large files download only when needed.
Think of it like a library card catalog: instead of storing every book on every shelf, you store a reference card that points to where the real book lives in storage.
When to Use Git LFS
Use Git LFS for:
- Datasets over ~10 MB that change occasionally
- Trained model files (.pkl, .h5, .pt files)
- Database snapshots or compressed archives
- Binary assets that must be versioned with code
Don't use it for constantly changing files or anything already handled by cloud storage or artifact registries.
Basic LFS Workflow
After installing Git LFS, you track file patterns:
git lfs track "*.csv"
git lfs track "models/*.pkl"
This creates a .gitattributes file marking which files use LFS. Then commit normally—LFS handles the rest automatically.
Key Takeaway: Git LFS lets you version control large data assets without bloating your repository, storing big files separately while keeping lightweight pointers in Git history.