Best Function Calling Small LLM: A Practical Shortlist

"Small" in the context of LLMs usually means anything from 1B to around 14B parameters — models you can run on a single GPU, at the edge, or cheaply behind an API. The honest answer to "what's the best function calling small LLM" is: it depends on whether you need self-hosted open weights or a hosted small model with production-grade reliability, because those are two very different shortlists with different failure modes.

If you're picking an open-weight model to self-host, your best current options are Qwen2.5-7B/14B-Instruct, Llama 3.1 8B Instruct, Mistral Nemo 12B, and Hermes-3 8B (fine-tuned specifically for tool calling). If you want a hosted small model with strong, tested tool-use behavior and no infrastructure to manage, a fast model like Claude Haiku accessed through an API layer is usually the more practical choice, especially once you factor in JSON reliability and multi-turn tool loops.

What makes a small model good at function calling

Parameter count alone doesn't predict tool-use quality. The things that actually matter:

• Structured output consistency — does it reliably emit valid JSON matching your schema, turn after turn, without drifting into prose?
• Parallel tool calls — can it call two or three tools in one turn when the task requires it, or does it serialize everything?
• Argument accuracy — does it hallucinate parameters that don't exist in your schema, or guess types incorrectly?
• Multi-step tool loops — after receiving a tool result, does it correctly decide whether to call another tool or respond to the user?
• Refusal to call unnecessary tools — small models are more prone to calling a tool when a direct answer would do.

Benchmarks like the Berkeley Function-Calling Leaderboard (BFCL) are the closest thing to a standardized comparison, and they're worth checking before committing to a model, because rankings shift with every release.

Top small open-weight models for function calling

Qwen2.5-7B-Instruct / 14B-Instruct Currently one of the strongest small open models for structured tool use. Handles parallel calls reasonably well and has native support for OpenAI-style tool schemas in most serving frameworks (vLLM, Ollama, llama.cpp).

Llama 3.1 8B Instruct Meta trained this generation specifically with tool-calling data. It's solid for single-tool-per-turn workflows but degrades faster than larger models on multi-step chains or ambiguous schemas.

Mistral Nemo 12B / Mistral 7B v0.3 Good JSON discipline, reasonable latency, widely supported in inference stacks. Nemo is the better pick if you can afford the extra parameters.

Hermes-3 (Llama 3.1 8B fine-tune) Built by Nous Research with heavy emphasis on function calling and agentic behavior. Worth testing if the base Llama model isn't reliable enough for your schema complexity.

Functionary (Meetkai) Purpose-built for function calling from the ground up, with a custom prompt format optimized for tool selection. Smaller community than the models above, but consistently competitive on BFCL for its size class.

None of these match a frontier model's reasoning depth, so if your tools require multi-hop planning (call A, interpret result, decide between B or C, then call D), expect more retries and validation errors than with a larger model.

When a hosted small model is the better call

Self-hosting gives you control over weights and cost per token, but it also means you own prompt engineering, JSON validation, retry logic, and infrastructure. For many teams, a hosted small-but-capable model gets to production faster and more reliably.

This is where SubToAPI fits: it turns your existing Claude access into a standard HTTPS API with application keys (sub_live_...), streaming, and structured tool use, so you can point your app at a fast Claude model without running your own inference stack. Claude's tool-use format handles schema validation, multi-turn tool loops, and parallel calls out of the box — which is exactly the part that's hardest to get right with small self-hosted models.

A basic tool-calling request looks like this:

curl https://api.subtoapi.app/v1/messages \
  -H "Authorization: Bearer $SUBTOAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-haiku",
    "max_tokens": 512,
    "tools": [{
      "name": "get_weather",
      "description": "Get current weather for a location",
      "input_schema": {
        "type": "object",
        "properties": {
          "location": {"type": "string"}
        },
        "required": ["location"]
      }
    }],
    "messages": [
      {"role": "user", "content": "What is the weather in Lisbon?"}
    ]
  }'

The response comes back with a tool_use block containing structured arguments you can execute and feed back into the conversation — no custom parsing needed. See the tools docs for the full request/response shape and the quickstart to get an API key from your existing Claude subscription.

Practical tips regardless of which model you pick

• Keep schemas narrow. Small models struggle more with large tool catalogs. If you have 15+ tools, group them and let the model pick a category first.
• Validate everything. Even good models occasionally emit malformed JSON — always parse defensively and retry on failure.
• Test with real user phrasing. Benchmarks use clean prompts; your users won't. Run your own eval set before trusting a model in production.
• Watch latency, not just accuracy. A model that's 2% more accurate but 3x slower often isn't worth it in a tool loop that may call several times per turn.
• Start with streaming disabled for tool calls, then add streaming once your tool-use logic is stable — debugging partial tool-call JSON mid-stream is harder than debugging complete responses.

Bottom line

For self-hosted, open-weight function calling, Qwen2.5 and Hermes-3 are the strongest small-model picks right now. For teams that want reliable tool use without managing inference infrastructure, a hosted small Claude model is often the faster path to production — check pricing if you want to compare the cost of hosting your own small model against a hosted API.

FAQ

Is a 7B model good enough for production function calling? For simple, single-tool workflows with narrow schemas, yes. For multi-step tool chains or large tool catalogs, you'll likely see more JSON errors and incorrect tool selection than with a larger or hosted model.

What's the difference between function calling and tool use? They're the same concept under different naming — "function calling" is OpenAI's original term, "tool use" is the more general/Anthropic term. Both describe a model returning structured arguments to invoke an external function.

Do I need to fine-tune a small model for function calling? Not necessarily. Models like Qwen2.5, Llama 3.1, and Hermes-3 are already instruction-tuned for tool use. Fine-tuning helps if your schemas are unusual or you need very high accuracy on a narrow domain.