Course contentsShow
← AI Engineering
Lesson 208 of 1,8866. Embeddings and Semantic SearchFree lesson

Token vs Sentence vs Document Embeddings

Differentiate between embedding granularities: individual tokens, full sentences, and longer documents.

Token vs Sentence vs Document Embeddings

What you'll learn: How embeddings can represent different units of text—individual tokens, complete sentences, or entire documents—and when to use each.

The Core Concept

Embeddings can capture meaning at different granularities. Just as you can describe a book by its individual words, its chapters, or the whole story, embeddings can represent:

  • Token embeddings: Vectors for single words or subwords (like "cat" or "##ing")
  • Sentence embeddings: Vectors for complete sentences or phrases
  • Document embeddings: Vectors for paragraphs, articles, or entire documents

Each level captures meaning appropriate to its scope.

Understanding the Differences

Token embeddings represent the smallest units. They're useful when you need word-level understanding—like detecting which specific words appear in text or analyzing individual terms. Think of them as capturing "vocabulary."

Sentence embeddings represent complete thoughts. They capture the meaning of an entire sentence as a single vector, considering word order and grammar. These are ideal for comparing questions, finding similar statements, or semantic search at the sentence level. The sentence "Dogs chase cats" has a different meaning than "Cats chase dogs," even though they contain the same tokens.

Document embeddings compress longer texts into single vectors. They capture themes, topics, and overall intent across multiple sentences. Use these when comparing articles, categorizing documents, or finding similar blog posts.

Choosing the Right Granularity

The granularity you choose depends on your task. Searching for similar questions? Use sentence embeddings. Finding related articles? Use document embeddings. Building a vocabulary-based feature? Use token embeddings.

Key Takeaway: Embeddings exist at multiple levels—tokens capture individual words, sentences capture complete thoughts, and documents capture overall meaning. Choose the granularity that matches your comparison or search task.