Limitations of Embeddings
You'll learn: Why embeddings, despite their power, have real constraints that can trip up even well-designed systems.
Core Idea
Embeddings transform text into numbers, but they're not magic. They face three critical limitations that can quietly break your semantic search or similarity features:
1. Out-of-Vocabulary (OOV) Terms
When embeddings encounter words they've never seen during training—new slang, brand names, technical jargon, or typos—they struggle. Some models break unknown words into subwords, but meaning can still degrade. It's like asking someone to describe a concept in a language where they don't know the key vocabulary.
2. Domain Shift
Embeddings trained on general text (news, books, web pages) may fail when applied to specialized domains like medical records, legal contracts, or gaming forums. The word "cell" means something very different in biology vs spreadsheets vs prisons. An embedding model trained on Wikipedia won't capture these nuances unless it saw similar domain-specific text.
3. Fixed Maximum Sequence Length
Every embedding model has a hard token limit—often 512 or 8192 tokens. Feed it a 50-page document, and you must truncate, chunk, or summarize. You lose context and potentially critical information that falls outside the window.
Why This Matters
If you're building semantic search over niche technical documentation with lots of product codes (OOV), or analyzing century-old legal texts (domain shift), or indexing long research papers (length limits), your embeddings may silently underperform. Recognizing these weaknesses helps you choose preprocessing strategies, specialized models, or hybrid approaches.
Key Takeaway: Embeddings fail gracefully but predictably when facing unknown vocabulary, unfamiliar domains, or text exceeding their maximum length—know these limits before they surprise you in production.