Course contentsShow
← Machine Learning and Deep Learning
Lesson 1129 of 3,53825. Word Embeddings and Contextual RepresentationsFree lesson

FastText and Subword Embeddings

Learn how FastText improves on Word2Vec by representing words as bags of character n-grams for better rare word handling.

FastText and Subword Embeddings

What you'll learn: How FastText extends Word2Vec by breaking words into character-level pieces to handle rare and unseen words more effectively.

The Problem with Word2Vec

Word2Vec (which you learned earlier) treats each word as an atomic unit—a single indivisible token. This creates a problem: what happens when you encounter "unbelievable" but your model only knows "believe"? Or misspellings like "gooogle"? Word2Vec has no way to leverage the similarity between these related words.

FastText's Core Innovation

FastText represents each word as a bag of character n-grams instead of a single unit. Think of it like breaking "where" into puzzle pieces: <wh, whe, her, ere, re> (plus the full word <where>). The special brackets < > mark word boundaries.

Each n-gram gets its own embedding vector, and a word's final embedding is the sum of all its n-gram embeddings. This means "unbelievable" shares subword components with "believe" (bel, eli, lie, etc.), allowing the model to infer meaning even for rare words.

Why This Matters

  • Rare words: Even if "antiestablishment" appears once, its pieces (anti, esta, lish, etc.) likely appear in many common words
  • Misspellings: "gooogle" shares most n-grams with "google"
  • Morphology: Languages like German or Turkish with complex word formation benefit hugely
  • Out-of-vocabulary: You can generate embeddings for words never seen during training

The Analogy

Imagine learning a language by memorizing whole phrases (Word2Vec) versus learning the building blocks—prefixes, roots, suffixes (FastText). The second approach lets you understand new combinations you've never heard before.

Key Takeaway: FastText improves on Word2Vec by representing words as collections of character n-grams, enabling better handling of rare words, morphological variants, and even out-of-vocabulary terms through shared subword information.