Course contentsShow
← AI Engineering
Lesson 1243 of 1,88630. Monitoring and ObservabilityFree lesson

Understanding Distribution Drift in LLM Systems

Learn what drift means for AI systems: input drift, output drift, and concept drift in production environments.

Understanding Distribution Drift in LLM Systems

What you'll learn: You'll understand the three types of drift that can quietly degrade your LLM system's performance over time.

The Core Idea

Distribution drift occurs when the statistical properties of your data change between training and production, or over time in production. For LLM systems, this manifests in three critical ways:

Input Drift happens when the prompts users send start looking different from what you expected. Maybe your restaurant chatbot was designed for dinner reservations, but users increasingly ask about delivery options. The vocabulary, intent distribution, or query complexity shifts.

Output Drift occurs when your model's responses change character over time, even with similar inputs. This can happen due to model updates, temperature variations in sampling, or cascade effects in multi-step chains where small changes compound.

Concept Drift is the most subtle: the relationship between inputs and correct outputs changes. What was a "good" answer six months ago might be outdated now. A travel assistant trained pre-pandemic would have concept drift about international travel requirements.

Why This Matters

Think of drift like continental drift—imperceptibly slow but fundamentally changing the landscape. Your carefully tuned prompts and evaluation baselines can become obsolete without obvious warnings. The metrics you tracked in lesson 1232 might stay stable while user satisfaction drops because what "quality" means has shifted.

Drift is why static benchmarks aren't enough. The token usage patterns you monitored in lesson 1177 and the performance baselines from lesson 1167 need continuous validation against evolving real-world distributions.

Key Takeaway: Distribution drift—input patterns changing, output characteristics shifting, or the input-output relationship evolving—is inevitable in production LLM systems and requires active detection to maintain quality.