Data Quality Over Quantity
You'll learn why 100 carefully curated examples often beat 10,000 messy ones when fine-tuning LLMs.
The Core Principle
When preparing data for fine-tuning, your instinct might be to gather everything—scraping thousands of examples from logs, databases, or user interactions. But here's the counterintuitive truth: a small set of pristine, representative examples typically outperforms a massive dataset filled with noise, inconsistencies, or irrelevant patterns.
Think of fine-tuning like teaching a chef. Would you rather show them 100 perfectly executed recipes with clear techniques, or 10,000 random kitchen experiments—some brilliant, many mediocre, some outright failures? The model learns from patterns, and noisy data teaches confused, contradictory patterns.
Why Quality Wins
Noise amplifies bad behaviors. If your 10,000 examples include:
- Inconsistent formatting (some JSON, some plain text)
- Factual errors or hallucinations from production logs
- Off-topic conversations or edge cases
- Conflicting instructions or tone shifts
...the model learns to replicate that chaos. It might hallucinate more, format inconsistently, or adopt the wrong tone.
High-quality examples are:
- Accurate: Verified ground truth, not raw user data
- Consistent: Uniform format, tone, and structure
- Representative: Cover key use cases without redundancy
- Clean: Free from typos, artifacts, or irrelevant context
A focused 100-example dataset ensures every training step reinforces the exact behavior you want, without dilution or contradiction.
Key Takeaway: Invest time in curating, validating, and cleaning a small golden dataset rather than blindly scaling up noisy training data—your fine-tuned model's reliability depends on it.