Understanding Distribution Drift in LLM Systems
What you'll learn: You'll understand the three types of drift that can quietly degrade your LLM system's performance over time.
The Core Idea
Distribution drift occurs when the statistical properties of your data change between training and production, or over time in production. For LLM systems, this manifests in three critical ways:
Input Drift happens when the prompts users send start looking different from what you expected. Maybe your restaurant chatbot was designed for dinner reservations, but users increasingly ask about delivery options. The vocabulary, intent distribution, or query complexity shifts.
Output Drift occurs when your model's responses change character over time, even with similar inputs. This can happen due to model updates, temperature variations in sampling, or cascade effects in multi-step chains where small changes compound.
Concept Drift is the most subtle: the relationship between inputs and correct outputs changes. What was a "good" answer six months ago might be outdated now. A travel assistant trained pre-pandemic would have concept drift about international travel requirements.
Why This Matters
Think of drift like continental drift—imperceptibly slow but fundamentally changing the landscape. Your carefully tuned prompts and evaluation baselines can become obsolete without obvious warnings. The metrics you tracked in lesson 1232 might stay stable while user satisfaction drops because what "quality" means has shifted.
Drift is why static benchmarks aren't enough. The token usage patterns you monitored in lesson 1177 and the performance baselines from lesson 1167 need continuous validation against evolving real-world distributions.
Key Takeaway: Distribution drift—input patterns changing, output characteristics shifting, or the input-output relationship evolving—is inevitable in production LLM systems and requires active detection to maintain quality.