Embedding-Based Drift Detection
What you'll learn: How to use embedding spaces to efficiently detect when user queries or inputs semantically shift over time.
The Core Idea
Statistical drift detection (which you've just learned) works well for numeric features, but LLM systems deal with text. Embedding-based drift detection converts text into dense vector representations (embeddings), then monitors changes in the geometric distribution of these vectors over time.
Think of it like tracking a flock of birds migrating. Instead of checking individual bird positions (statistical features of text), you represent each bird as a point in 3D space and watch whether the flock's overall shape and center shifts. If your users suddenly start asking about entirely different topics, the "shape" of your embedding cluster will change noticeably.
How It Works
You maintain a reference distribution of embeddings from your baseline period (e.g., last month's queries). For new incoming data, you:
- Embed each query using the same embedding model
- Compare distributions using distance metrics between embedding clusters
- Detect drift when the new distribution deviates significantly
Common techniques include tracking:
- Centroid distance: How far the average new embedding drifts from baseline
- Cluster overlap: Whether new embeddings form separate clusters
- Cosine similarity distributions: Changes in typical similarity scores between queries
Why This Matters
Embedding-based detection catches semantic drift that statistical methods miss. If users shift from asking "refund policy" to "exchange procedures," the words differ but traditional statistics might not flag it. Embeddings capture that these are related but distinct topics.
Key Takeaway: Embedding-based drift detection monitors semantic shifts by comparing the geometric distribution of vector representations, catching meaningful changes in user intent that word-level statistics alone would miss.