Course contentsShow
← AI Engineering
Lesson 410 of 1,88610. RAG - Improving RetrievalFree lesson

LLM-Based Relevance Scoring

Using language models to judge document relevance when human annotations are unavailable.

LLM-Based Relevance Scoring

What You'll Learn

You'll learn how to use language models themselves to evaluate whether retrieved documents are relevant to a query when you don't have human-labeled test data.

The Core Idea

When evaluating retrieval quality, you ideally need ground truth: human experts marking which documents are relevant to each query. But creating these annotations is expensive and time-consuming. Instead, you can ask an LLM to act as the judge.

LLM-based relevance scoring means prompting a language model to evaluate whether a retrieved document answers or relates to a given query. You provide the query and document as context, then ask the LLM to score relevance (often binary yes/no, or on a scale like 0-5).

Think of it like having a tireless research assistant who reads every query-document pair and gives you an honest opinion: "Does this document help answer this question?"

How It Works in Practice

You craft a prompt like:

Given the query: "What causes ocean tides?"
And the document: "[retrieved text...]"
Is this document relevant? Answer: Yes or No

The LLM analyzes semantic meaning, context, and content overlap—often catching nuances that simple keyword matching misses. You can collect these scores across your test set to calculate metrics like precision and recall without manual labeling.

Limitations to Keep in Mind

LLMs aren't perfect judges. They can hallucinate, be overly generous, or miss domain-specific nuances. Still, they're faster and cheaper than human annotation for initial evaluation rounds, and correlate reasonably well with human judgments.

Key Takeaway: LLM-based relevance scoring lets you automatically evaluate retrieval quality by having the model judge document relevance, providing a scalable alternative to expensive human annotations.