Course contentsShow
← AI Engineering
Lesson 1245 of 1,88630. Monitoring and ObservabilityFree lesson

Embedding-Based Drift Detection

Use embedding spaces to monitor semantic drift in user queries and detect distribution shifts efficiently.

Embedding-Based Drift Detection

What you'll learn: How to use embedding spaces to efficiently detect when user queries or inputs semantically shift over time.

The Core Idea

Statistical drift detection (which you've just learned) works well for numeric features, but LLM systems deal with text. Embedding-based drift detection converts text into dense vector representations (embeddings), then monitors changes in the geometric distribution of these vectors over time.

Think of it like tracking a flock of birds migrating. Instead of checking individual bird positions (statistical features of text), you represent each bird as a point in 3D space and watch whether the flock's overall shape and center shifts. If your users suddenly start asking about entirely different topics, the "shape" of your embedding cluster will change noticeably.

How It Works

You maintain a reference distribution of embeddings from your baseline period (e.g., last month's queries). For new incoming data, you:

  1. Embed each query using the same embedding model
  2. Compare distributions using distance metrics between embedding clusters
  3. Detect drift when the new distribution deviates significantly

Common techniques include tracking:

  • Centroid distance: How far the average new embedding drifts from baseline
  • Cluster overlap: Whether new embeddings form separate clusters
  • Cosine similarity distributions: Changes in typical similarity scores between queries

Why This Matters

Embedding-based detection catches semantic drift that statistical methods miss. If users shift from asking "refund policy" to "exchange procedures," the words differ but traditional statistics might not flag it. Embeddings capture that these are related but distinct topics.

Key Takeaway: Embedding-based drift detection monitors semantic shifts by comparing the geometric distribution of vector representations, catching meaningful changes in user intent that word-level statistics alone would miss.