Course contentsShow
← Machine Learning and Deep Learning
Lesson 1331 of 3,53829. Core NLP TasksFree lesson

Embedding Dimensionality and Normalization

Why embeddings are typically 384-1024 dimensions and how L2 normalization enables cosine similarity via dot product.

Embedding Dimensionality and Normalization

What you'll learn: Why sentence embeddings typically range from 384-1024 dimensions and how L2 normalization enables efficient similarity comparisons.

Why 384-1024 Dimensions?

Embedding dimensions represent the "information capacity" of your vector representation. Think of it like a filing cabinet: more drawers (dimensions) mean you can organize and distinguish between more nuanced semantic concepts.

Too few dimensions (e.g., 64-128): Can't capture subtle semantic differences. Like trying to describe every book in a library with only ten adjectives—you lose important distinctions.

Too many dimensions (e.g., 2048+): Computationally expensive with diminishing returns. You're adding empty drawers that don't improve organization much.

The 384-1024 sweet spot balances expressiveness and efficiency. Models like all-MiniLM-L6-v2 use 384 dimensions, while more powerful models like all-mpnet-base-v2 use 768. Research shows this range captures semantic relationships effectively while remaining practical for real-world applications.

L2 Normalization: The Similarity Shortcut

L2 normalization scales each embedding vector to unit length (length = 1). This is like adjusting all vectors to lie on the surface of a sphere.

The magic: When vectors are L2-normalized, cosine similarity (which measures angle between vectors) becomes mathematically equivalent to a simple dot product.

Without normalization: cosine_sim(A, B) = (A · B) / (||A|| × ||B||)

With normalization: cosine_sim(A, B) = A · B (since ||A|| = ||B|| = 1)

This means you can use fast matrix multiplication instead of expensive cosine calculations when finding similar embeddings—critical when searching through millions of vectors.

Key takeaway: Embedding dimensions between 384-1024 provide optimal semantic capacity for most NLP tasks, and L2 normalization enables fast similarity searches by reducing cosine similarity to a simple dot product operation.