Evaluation Metrics for Agent Planning
What you'll learn: How to measure whether your AI agent's plans are actually good using concrete metrics.
Why Measuring Plan Quality Matters
When your agent creates a plan to solve a task, you need objective ways to judge its performance. Just like grading a student's homework, you need clear criteria—not just "it worked" or "it didn't."
Core Planning Metrics
Success Rate
The most fundamental metric: did the agent complete the goal? This is your pass/fail measurement across multiple attempts. A 90% success rate means the agent succeeded in 9 out of 10 trials.
Efficiency (Computational Cost)
How many resources did the plan consume? This includes:
- Time taken to generate and execute the plan
- Token usage (for LLM-based planning)
- API calls made during planning and execution
Think of this like measuring fuel efficiency—did your agent take the highway or get lost on backroads?
Number of Steps
How many actions did the agent take to reach the goal? A shorter plan is often better (assuming it succeeds). If one agent needs 3 steps and another needs 12 to book the same flight, the first is more efficient.
Robustness to Failures
How well does the plan handle unexpected obstacles? Test this by:
- Introducing random tool failures
- Providing incomplete information
- Changing conditions mid-execution
A robust agent recovers gracefully; a brittle one collapses at the first hiccup.
Combining Metrics
Real-world evaluation typically combines these: A plan with 100% success but 1000 steps might be worse than 95% success with 10 steps, depending on your use case.
Key Takeaway: Effective agent planning evaluation requires multiple metrics—success rate (did it work?), efficiency (how fast/cheap?), step count (how simple?), and robustness (how resilient?)—to fully understand performance quality.