Course contentsShow
← AI Engineering
Lesson 1244 of 1,88630. Monitoring and ObservabilityFree lesson

Statistical Methods for Detecting Input Drift

Apply KL divergence, JS divergence, and chi-square tests to detect changes in prompt distributions over time.

Statistical Methods for Detecting Input Drift

What You'll Learn

You'll learn how to apply KL divergence, JS divergence, and chi-square tests to quantitatively detect when your LLM's incoming prompt distributions change over time.

The Core Idea

In lesson 1243, you learned why input drift matters—users change how they phrase questions, new topics emerge, and seasonal patterns shift. Now you need statistical tools to catch these changes automatically.

Think of it like a security system: instead of manually watching camera feeds 24/7, you install motion detectors. These statistical methods are your "drift detectors" that alert you when input patterns deviate significantly from your baseline.

Three Key Methods

KL Divergence (Kullback-Leibler)

Measures how one probability distribution differs from a reference distribution. Perfect for comparing current week's prompt features against your baseline. It's asymmetric: comparing A to B gives different results than B to A.

When to use: You have a clear "reference" distribution (like training data or first month of production).

JS Divergence (Jensen-Shannon)

A symmetric version of KL divergence—bounded between 0 and 1. It treats both distributions equally, making it more intuitive for drift detection.

When to use: You want a single, interpretable score where 0 = identical and 1 = completely different.

Chi-Square Test

Tests whether observed categorical distributions match expected ones. Excellent for discrete features like prompt length buckets, topic categories, or user segments.

When to use: Your prompt features are categorical (e.g., question type, language, department).

Practical Application

You'd compute these metrics daily or weekly between your current input distribution and your baseline. Set thresholds (e.g., JS divergence > 0.15) to trigger alerts, then investigate what changed—new user behavior, seasonal trends, or system issues.

Key Takeaway: KL/JS divergence and chi-square tests transform vague "something feels different" intuitions into concrete, measurable drift scores that trigger retraining, prompt updates, or further investigation.