Semantic Similarity Metrics
What You'll Learn
You'll learn how to measure whether LLM-generated text preserves the meaning of reference text, even when the words are completely different.
Why Surface-Level Matching Isn't Enough
Traditional metrics like exact string matching or even BLEU (which counts n-gram overlaps) fail when your LLM generates semantically correct but differently-worded output. If your reference answer is "The cat sat on the mat" and your LLM outputs "A feline rested on the rug," word-matching metrics score this poorly—yet the meaning is nearly identical.
Analogy: It's like judging whether two people said the same thing by only checking if they used identical words, rather than understanding their actual message.
Embedding-Based Similarity
This approach converts both the reference and generated text into embedding vectors (numerical representations that capture meaning), then measures how close these vectors are using cosine similarity.
- Scores range from -1 to 1 (typically 0 to 1 for text)
- Higher scores mean more semantic similarity
- Captures synonyms, paraphrasing, and conceptual overlap
- Fast to compute once embeddings are generated
BERTScore
BERTScore takes this further by using contextual embeddings (like those from BERT) to compare texts token by token, then aggregating the best matches.
- Computes precision, recall, and F1 between token embeddings
- Handles word reordering better than simple similarity
- More nuanced than whole-text embedding comparison
- Correlates well with human judgment of semantic equivalence
Both metrics help you evaluate whether your LLM's output is meaningfully correct rather than just word-for-word identical—crucial for real-world applications where there are many valid ways to express the same idea.
Key Takeaway: Semantic similarity metrics use embeddings to measure meaning preservation, allowing you to evaluate LLM outputs that are semantically correct but textually different from reference answers.