Course contentsShow
← Machine Learning and Deep Learning
Lesson 2096 of 3,53844. AI Agents: FoundationsFree lesson

Evaluation Metrics for Agent Planning

Measuring plan quality: success rate, efficiency, number of steps, and robustness to failures.

Evaluation Metrics for Agent Planning

What you'll learn: How to measure whether your AI agent's plans are actually good using concrete metrics.

Why Measuring Plan Quality Matters

When your agent creates a plan to solve a task, you need objective ways to judge its performance. Just like grading a student's homework, you need clear criteria—not just "it worked" or "it didn't."

Core Planning Metrics

Success Rate

The most fundamental metric: did the agent complete the goal? This is your pass/fail measurement across multiple attempts. A 90% success rate means the agent succeeded in 9 out of 10 trials.

Efficiency (Computational Cost)

How many resources did the plan consume? This includes:

  • Time taken to generate and execute the plan
  • Token usage (for LLM-based planning)
  • API calls made during planning and execution

Think of this like measuring fuel efficiency—did your agent take the highway or get lost on backroads?

Number of Steps

How many actions did the agent take to reach the goal? A shorter plan is often better (assuming it succeeds). If one agent needs 3 steps and another needs 12 to book the same flight, the first is more efficient.

Robustness to Failures

How well does the plan handle unexpected obstacles? Test this by:

  • Introducing random tool failures
  • Providing incomplete information
  • Changing conditions mid-execution

A robust agent recovers gracefully; a brittle one collapses at the first hiccup.

Combining Metrics

Real-world evaluation typically combines these: A plan with 100% success but 1000 steps might be worse than 95% success with 10 steps, depending on your use case.

Key Takeaway: Effective agent planning evaluation requires multiple metrics—success rate (did it work?), efficiency (how fast/cheap?), step count (how simple?), and robustness (how resilient?)—to fully understand performance quality.