Course contentsShow
← AI Engineering
Lesson 1266 of 1,88631. LLM Observability PlatformsFree lesson

LangSmith Evaluations and Metrics

Run evaluations against datasets, configure custom evaluators, and track metrics like latency, cost, and accuracy over time.

LangSmith Evaluations and Metrics

What you'll learn: How to systematically measure and improve your LLM application's quality using LangSmith's evaluation framework.

Understanding LangSmith Evaluations

Now that you've captured traces and created datasets in LangSmith, it's time to put them to work. Evaluations let you run your LLM application against a curated dataset and measure how well it performs. Think of it like a standardized test for your AI system—you provide the questions (your dataset), and LangSmith grades the answers.

LangSmith supports running evaluations against the datasets you've created, applying evaluators that check correctness, relevance, tone, or any custom criteria you define. This bridges the gap between "it seems to work" and "I have data proving it works."

Configuring Custom Evaluators

Out-of-the-box metrics like exact match are useful, but real applications need custom evaluators. You can create evaluators that:

  • Check if outputs contain required information
  • Validate JSON structure or API response format
  • Compare semantic similarity between expected and actual answers
  • Use LLM-as-judge patterns to assess quality dimensions

These evaluators run automatically across your entire dataset, generating consistent, repeatable measurements.

Tracking Metrics Over Time

LangSmith doesn't just show you today's scores—it tracks metrics across evaluation runs. Monitor:

  • Accuracy: How often outputs match expectations
  • Latency: Average response time per request
  • Cost: Token usage and API expenses per evaluation

By comparing runs side-by-side, you can see whether a prompt change improved accuracy, whether a new model increased cost, or whether performance degraded after deployment.

Key Takeaway: LangSmith evaluations transform subjective judgment into measurable data by running automated tests against datasets, using custom evaluators, and tracking accuracy, latency, and cost metrics over time—giving you confidence that changes improve your system.