The Curse of Dimensionality
You'll learn: Why searching through high-dimensional vector spaces becomes surprisingly difficult and computationally expensive.
The Core Problem
When embeddings have hundreds or thousands of dimensions (like OpenAI's 1,536-dimensional vectors), something counterintuitive happens: all points start looking equally distant from each other. This is the "curse of dimensionality."
Why High Dimensions Are Costly
Imagine searching for the nearest neighbor in a 2D space—you can visualize it easily. Now picture 1,536 dimensions. To find similar vectors, you must:
- Calculate distance to every vector in your database (cosine similarity or dot product)
- Compare across all 1,536 dimensions for each calculation
- Sort results to find the top-k matches
With a million documents, that's millions of calculations × 1,536 dimensions each. The compute grows linearly with both dimensions and dataset size.
The Distance Problem
In high dimensions, the difference between the "nearest" and "farthest" neighbor shrinks dramatically. Most points cluster in a thin shell at similar distances from any query point. This makes distinguishing truly relevant results harder and requires examining more candidates.
How Vector DBs Optimize
Vector databases like Pinecone, Weaviate, and Qdrant use specialized algorithms (which we'll cover later) to avoid checking every single vector. They trade perfect accuracy for massive speed improvements—often finding the approximate nearest neighbors in milliseconds instead of seconds.
Key Takeaway: High-dimensional vector search is expensive because distance calculations scale with dimensions and dataset size, and points become harder to distinguish—vector databases use clever approximation techniques to make this practical.