Course contentsShow
← AI Engineering
Lesson 799 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Semantic Similarity Metrics

Measuring embedding-based similarity and BERTScore to evaluate meaning preservation beyond surface-level text matching.

Semantic Similarity Metrics

What You'll Learn

You'll learn how to measure whether LLM-generated text preserves the meaning of reference text, even when the words are completely different.

Why Surface-Level Matching Isn't Enough

Traditional metrics like exact string matching or even BLEU (which counts n-gram overlaps) fail when your LLM generates semantically correct but differently-worded output. If your reference answer is "The cat sat on the mat" and your LLM outputs "A feline rested on the rug," word-matching metrics score this poorly—yet the meaning is nearly identical.

Analogy: It's like judging whether two people said the same thing by only checking if they used identical words, rather than understanding their actual message.

Embedding-Based Similarity

This approach converts both the reference and generated text into embedding vectors (numerical representations that capture meaning), then measures how close these vectors are using cosine similarity.

  • Scores range from -1 to 1 (typically 0 to 1 for text)
  • Higher scores mean more semantic similarity
  • Captures synonyms, paraphrasing, and conceptual overlap
  • Fast to compute once embeddings are generated

BERTScore

BERTScore takes this further by using contextual embeddings (like those from BERT) to compare texts token by token, then aggregating the best matches.

  • Computes precision, recall, and F1 between token embeddings
  • Handles word reordering better than simple similarity
  • More nuanced than whole-text embedding comparison
  • Correlates well with human judgment of semantic equivalence

Both metrics help you evaluate whether your LLM's output is meaningfully correct rather than just word-for-word identical—crucial for real-world applications where there are many valid ways to express the same idea.

Key Takeaway: Semantic similarity metrics use embeddings to measure meaning preservation, allowing you to evaluate LLM outputs that are semantically correct but textually different from reference answers.