Defining Pass/Fail Criteria for Regression Tests
What you'll learn: How to set clear thresholds that determine whether your LLM system passes or fails regression tests.
Why Binary Decisions Matter
Once you've automated your regression suite and collected outputs, you need a systematic way to decide: did this test pass or fail? Without clear criteria, every test run requires manual judgment calls — defeating the purpose of automation.
Three Common Threshold Approaches
Exact Match Thresholds
For deterministic outputs (structured data, classifications), you might require 100% exact match. If even one field differs from ground truth, the test fails.
Example: A function-calling test expects {"action": "book_flight", "destination": "LAX"} — anything else fails.
Semantic Similarity Thresholds
For natural language outputs, you calculate semantic similarity (like cosine similarity between embeddings) and set a minimum score.
Example: Require similarity ≥ 0.85 to ground truth. Outputs below this threshold fail, even if roughly correct.
LLM-as-Judge Score Thresholds
When using LLM-as-a-Judge evaluation, you define what scores constitute success. If your judge returns 1-5 ratings, you might say "pass = score ≥ 4."
Example: A judge evaluating helpfulness rates responses 1-5. You decide scores of 4 or 5 pass; 1-3 fail.
Setting Realistic Thresholds
Think of thresholds like quality control in manufacturing: too strict and every minor variation fails; too loose and defects slip through. Start conservative (stricter), then relax based on real-world performance patterns. Track false positives (incorrectly failed) and false negatives (incorrectly passed) to calibrate.
You can also use different thresholds for different test categories: critical accuracy tests might need 100% pass rates, while creative generation tests accept 90%.
Key Takeaway: Pass/fail criteria transform continuous evaluation scores into actionable binary decisions — choose thresholds that balance catching real regressions while tolerating acceptable variation in LLM outputs.