Testing and Debugging Function Definitions
What you'll learn: How to systematically test your function schemas with sample prompts and debug when the model misuses or misunderstands your tools.
Why Testing Matters
Even perfectly valid JSON schemas can confuse an LLM. The model might choose the wrong function, pass incorrect argument types, or miss required parameters entirely. Since you've defined your function registry and validated arguments (previous lessons), now you need to verify the LLM actually uses them correctly.
Testing Strategy
1. Create Test Prompts
Write prompts that should trigger each function. Test edge cases like ambiguous requests, missing information, or prompts that could match multiple functions.
test_cases = [
"What's the weather in Paris?", # Should call get_weather
"Tell me about the weather", # Missing required 'location'
"Send email to bob@test.com", # Should call send_email
"Get weather and send an email", # Parallel function test
]
2. Inspect Raw Responses
Log the model's tool_calls or function_call response before execution. Check if it chose the right function and provided sensible arguments.
3. Debug Common Issues
- Wrong function chosen: Your descriptions may overlap. Make them more distinct.
- Missing required params: The model might not understand what's required. Improve parameter descriptions or add examples.
- Wrong types: Add explicit type constraints in your schema and descriptions (e.g., "must be a string in YYYY-MM-DD format").
Iterative Refinement
Testing reveals where your descriptions are ambiguous. Refine them, retest, and build a regression test suite to prevent future breaks.
Key Takeaway: Testing function definitions with diverse prompts and inspecting raw model responses is essential to catch misunderstandings before execution—treat your function schemas like APIs that need comprehensive testing.