Creating Ground Truth from Historical Data
What you'll learn: How to turn real production logs, conversations, and user interactions into labeled examples you can use to evaluate your LLM system.
Why Historical Data is Your Goldmine
Ground truth doesn't have to be created from scratch. Your production systems have already generated thousands of real interactions—these are treasure troves of authentic examples that reflect actual user behavior, edge cases, and system performance in the wild.
The Extraction Process
Start by identifying valuable interactions in your logs:
- Production conversations where users explicitly expressed satisfaction or frustration
- Past interactions that were escalated to human review or support
- User feedback signals like thumbs up/down, ratings, or complaint tickets
- Successful completions where tasks were clearly finished
- Failed attempts that required retry or abandonment
Think of it like panning for gold: you're sifting through tons of raw material to find the nuggets worth keeping.
Labeling Your Examples
Once extracted, each example needs a judgment:
- Positive examples: Correct responses, successful task completions, helpful answers
- Negative examples: Hallucinations, policy violations, failed retrievals, incorrect classifications
- Partial success: Cases that got close but needed refinement
You can combine automated signals (user clicked "helpful") with manual review (an expert labels complex cases). The goal is honest labels that represent what "good" and "bad" actually look like for your specific use case.
Making It Actionable
Structure your ground truth with clear inputs and expected outputs. For a chatbot, that means: the user message(s), conversation context, the system's response, and your quality label. For classification, it's: the input text and the correct category.
Key Takeaway: Historical production data contains real-world examples that are more representative than synthetic test cases—extract meaningful interactions and label them systematically to build authentic ground truth datasets.