Course contentsShow
← Machine Learning and Deep Learning
Lesson 2082 of 3,53844. AI Agents: FoundationsFree lesson

Tool Use Evaluation Metrics

Measuring tool selection accuracy, parameter correctness, and overall tool use effectiveness in agents.

Tool Use Evaluation Metrics

What you'll learn: How to measure whether your AI agent is selecting the right tools, using them correctly, and achieving effective outcomes.

Why Measure Tool Use Quality?

Just because your agent can call tools doesn't mean it's doing so effectively. You need quantitative metrics to evaluate three critical dimensions: tool selection accuracy (did it pick the right tool?), parameter correctness (did it fill in the arguments properly?), and overall effectiveness (did it solve the task?).

Think of it like evaluating a chef: Did they choose the right utensil? Did they use the correct temperature settings? Did the dish turn out well? All three matter.

Core Metrics

Tool Selection Accuracy

Measures whether the agent chose the correct tool from its registry:

  • Accuracy: Percentage of cases where the selected tool matches the ground truth
  • Confusion matrix: Shows which tools get mistaken for others
  • Top-k accuracy: Whether the correct tool appears in the top k candidates

Parameter Correctness

Evaluates whether tool arguments are valid and semantically correct:

  • Exact match rate: Parameters match expected values exactly
  • Semantic equivalence: Parameters achieve the same intent (e.g., "NYC" vs "New York City")
  • Type correctness: Arguments match expected data types
  • Required field completion: All mandatory parameters provided

Overall Effectiveness

Measures end-to-end task success:

  • Task completion rate: Percentage of queries fully resolved
  • Tool call efficiency: Average number of tool calls needed
  • Error rate: How often tool execution fails
  • Execution time: Speed of tool use workflow

Creating Test Sets

You'll need labeled examples showing the correct tool, parameters, and expected outcome for representative queries—similar to creating ground truth for retrieval evaluation.

Key Takeaway: Effective agent evaluation requires measuring tool selection accuracy, parameter correctness, and task completion together—no single metric tells the whole story.