Tool Use Evaluation Metrics
What you'll learn: How to measure whether your AI agent is selecting the right tools, using them correctly, and achieving effective outcomes.
Why Measure Tool Use Quality?
Just because your agent can call tools doesn't mean it's doing so effectively. You need quantitative metrics to evaluate three critical dimensions: tool selection accuracy (did it pick the right tool?), parameter correctness (did it fill in the arguments properly?), and overall effectiveness (did it solve the task?).
Think of it like evaluating a chef: Did they choose the right utensil? Did they use the correct temperature settings? Did the dish turn out well? All three matter.
Core Metrics
Tool Selection Accuracy
Measures whether the agent chose the correct tool from its registry:
- Accuracy: Percentage of cases where the selected tool matches the ground truth
- Confusion matrix: Shows which tools get mistaken for others
- Top-k accuracy: Whether the correct tool appears in the top k candidates
Parameter Correctness
Evaluates whether tool arguments are valid and semantically correct:
- Exact match rate: Parameters match expected values exactly
- Semantic equivalence: Parameters achieve the same intent (e.g., "NYC" vs "New York City")
- Type correctness: Arguments match expected data types
- Required field completion: All mandatory parameters provided
Overall Effectiveness
Measures end-to-end task success:
- Task completion rate: Percentage of queries fully resolved
- Tool call efficiency: Average number of tool calls needed
- Error rate: How often tool execution fails
- Execution time: Speed of tool use workflow
Creating Test Sets
You'll need labeled examples showing the correct tool, parameters, and expected outcome for representative queries—similar to creating ground truth for retrieval evaluation.
Key Takeaway: Effective agent evaluation requires measuring tool selection accuracy, parameter correctness, and task completion together—no single metric tells the whole story.