Golden Datasets and Versioning
What You'll Learn
You'll learn how to create and maintain versioned collections of high-quality test cases that serve as stable benchmarks for detecting regressions when your LLM system changes.
The Core Idea
A golden dataset is your curated, trusted collection of test cases with known correct outputs. Think of it like a quality control checklist that you run every time you modify your system—whether you're changing prompts, swapping models, or updating retrieval logic.
The "golden" label means these examples represent the standard of quality you want to maintain. They're carefully selected (using your sampling strategies from lesson 823) and verified (through manual annotation from lesson 821), covering critical use cases, edge cases, and domain-specific scenarios (lesson 822).
Why Versioning Matters
Just as software code gets versioned, your golden dataset should too. Here's why:
Version control lets you:
- Track when examples were added or modified
- Understand which test cases applied to which system version
- Compare performance across dataset versions
- Roll back to previous datasets if needed
- Document why specific examples were included
Think of it like a photograph album of your system's expected behavior at different points in time.
Practical Approach
When you make a system change (new prompt, different model), run your golden dataset as a regression test. Use your task-specific metrics (lessons 796-805) and potentially LLM-as-a-judge (lessons 807-818) to score outputs against ground truth (lesson 819).
Version both the dataset itself and the evaluation results: golden_dataset_v1.2.json, eval_results_v1.2_gpt4.json. This creates an audit trail showing whether changes improved, maintained, or degraded performance.
Key Takeaway: Golden datasets are versioned, curated test collections that act as regression benchmarks—ensuring system changes don't break what already works well.