Function Calling with Local Models
You'll learn: How to enable function calling capabilities in open-source, locally-run language models through prompt engineering and fine-tuning techniques.
The Challenge with Local Models
Unlike OpenAI's API with native function calling support, most open-source models (Llama, Mistral, Phi, etc.) don't have built-in function calling. But you can still achieve this capability through two main approaches: prompt engineering and fine-tuning.
Prompt Engineering Approach
You can teach local models to generate function calls by crafting detailed system prompts that:
- Define available functions in natural language or JSON schema format
- Provide clear examples of when and how to call each function
- Specify the exact output format (typically JSON with function name and arguments)
- Include few-shot examples of correct function calling behavior
For example, your prompt might list functions, show example calls, and instruct: "When the user needs weather information, respond with {"function": "get_weather", "args": {"location": "..."}} and nothing else."
Fine-Tuning Approach
For production systems, fine-tuning works better. You create a dataset of conversations where the model correctly identifies when to call functions and formats them properly. This trains the model to:
- Recognize function-calling opportunities more reliably
- Generate consistent, valid function schemas
- Handle edge cases and ambiguous requests
Models like Functionary and Hermes are specifically fine-tuned for function calling and work well locally.
Key Takeaway: Local models can handle function calling through careful prompt engineering for quick prototypes or fine-tuning for robust production systems, giving you the power of tool use without depending on commercial APIs.