We built an internal tooling assistant that routes engineering queries to a suite of tools: Jira search, GitHub PR lookup, Confluence page retrieval, a Datadog metrics query, and a deployment history API. In staging, it handled everything correctly. In production with real queries from the engineering team, it started misfiring within the first day.
The model called the wrong tool. It passed arguments in the wrong format. It hallucinated tool names that did not exist. It called the GitHub tool with a Jira ticket ID as the repository parameter. It occasionally decided no tool was needed and answered from its training data instead, specifically for questions about our internal infrastructure.
None of these were model failures in the sense of the model being broken. They were the predictable behavior of a language model being used as a routing and dispatch layer without the production hardening that any routing layer needs. This post covers what we learned and the patterns we use now.
The Baseline Problem
LLM tool calling (where the model selects a tool from a list and generates a structured call with arguments) works well in demos because demos use clean queries, well-named tools, and happy-path inputs. Production does not.
The failure modes split into three categories:
Tool selection failures. The model picks the wrong tool. This happens most often when tool names or descriptions are ambiguous, when two tools have overlapping capability descriptions, or when the query contains vocabulary that activates the wrong tool's description.
Argument generation failures. The model picks the right tool but generates malformed arguments: wrong types, missing required fields, extra fields that the schema does not include, or values that are syntactically valid but semantically wrong (passing a user display name instead of a user ID, for example).
Execution decision failures. The model decides not to call any tool and answers from its training data, or calls a tool when it should compose an answer from prior context in the conversation.
All three are addressable. None require changing the underlying model.
Pattern 1: Constrained Tool Schemas
The most common source of argument failures is schemas that are too permissive. If a tool accepts string for a date parameter, the model may pass "last Monday", "2026-07-21", "July 21st", or "7/21", all of which your tool has to handle or reject.
The fix is to constrain schemas to the point where invalid values become structurally impossible:
# Permissive — invites malformed arguments
search_tool = {
"name": "search_issues",
"description": "Search Jira issues",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"date_from": {"type": "string", "description": "Start date"},
"status": {"type": "string"}
}
}
}
# Constrained — model cannot generate invalid values
search_tool = {
"name": "search_issues",
"description": "Search Jira issues by keyword. Returns issue keys, summaries, and assignees.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Search keywords. Do not include status or date filters here.",
"maxLength": 200
},
"date_from": {
"type": "string",
"description": "Filter to issues created on or after this date. Format: YYYY-MM-DD.",
"pattern": "^\\d{4}-\\d{2}-\\d{2}$"
},
"status": {
"type": "string",
"enum": ["open", "in_progress", "closed", "all"],
"description": "Filter by status. Use 'all' if status is not specified."
}
},
"required": ["query", "status"]
}
}
The constrained version adds three things: concrete format instructions in description fields, a regex pattern for the date, and an enum that eliminates free-form status values. We measured a substantial drop in argument validation errors after this kind of schema tightening, typically more than halved across all tools.
The description fields on individual properties matter more than the top-level tool description. The model reads them when generating arguments and uses them to make formatting decisions.
Pattern 2: Tool Call Validation Before Execution
Never pass a model-generated tool call directly to your execution layer. Validate the structure first.
from pydantic import BaseModel, field_validator
from datetime import datetime
from typing import Literal
import re
class SearchIssuesArgs(BaseModel):
query: str
date_from: str | None = None
status: Literal["open", "in_progress", "closed", "all"] = "all"
@field_validator("query")
@classmethod
def query_not_empty(cls, v: str) -> str:
if not v.strip():
raise ValueError("query cannot be empty")
return v.strip()
@field_validator("date_from")
@classmethod
def valid_date(cls, v: str | None) -> str | None:
if v is None:
return v
if not re.match(r"^\d{4}-\d{2}-\d{2}$", v):
raise ValueError(f"date_from must be YYYY-MM-DD, got: {v!r}")
datetime.strptime(v, "%Y-%m-%d") # raises ValueError if invalid date
return v
TOOL_VALIDATORS = {
"search_issues": SearchIssuesArgs,
# ... one validator per tool
}
def validate_tool_call(tool_name: str, arguments: dict) -> BaseModel:
validator = TOOL_VALIDATORS.get(tool_name)
if validator is None:
raise ValueError(f"Unknown tool: {tool_name!r}")
return validator(**arguments)
When validation fails, you have three options: retry the model with the validation error appended as context, fall back to a no-tool response, or return an error to the user. Which you choose depends on the tool and the cost of a retry. For our Jira search, a single retry with the validation error in the context resolves argument format failures most of the time.
Pattern 3: Retry With Structured Feedback
A failed tool call contains enough signal to guide a retry. Feed the failure reason back to the model as a system message rather than starting a new conversation:
async def call_with_retry(
client,
messages: list[dict],
tools: list[dict],
max_retries: int = 2,
) -> dict:
for attempt in range(max_retries + 1):
response = await client.messages.create(
model="claude-sonnet-5",
messages=messages,
tools=tools,
max_tokens=1024,
)
if response.stop_reason != "tool_use":
return response
tool_use = next(b for b in response.content if b.type == "tool_use")
try:
validated_args = validate_tool_call(tool_use.name, tool_use.input)
result = await execute_tool(tool_use.name, validated_args)
return result
except ValueError as e:
if attempt == max_retries:
raise
# Feed the validation error back as context
messages = messages + [
{"role": "assistant", "content": response.content},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": f"Validation error: {e}. Please retry with corrected arguments.",
"is_error": True,
}
],
},
]
raise RuntimeError("Max retries exceeded")
In our production system we measured first-attempt success rates around 91 percent and retry success around 97 percent for well-constrained schemas. The retry adds latency (one additional model call), so it is worth tracking retry rates per tool: a tool with a high retry rate has a schema or description problem, not a model problem.
Pattern 4: Tool Name Disambiguation
When two tools have overlapping capability, the model will sometimes pick the wrong one. The fix is almost never removing one of the tools. It is making the decision criteria explicit in the tool descriptions.
Bad:
"description": "Get information about a GitHub pull request"
"description": "Search GitHub pull requests"
Better:
"description": "Get full details for a single PR when you know the exact PR number. Input: repository name and PR number. Use this when the user references a specific PR by number (e.g. 'PR #1234')."
"description": "Search across all open PRs by keyword or author. Use this when the user does not know the PR number and is looking for PRs by topic, title substring, or author name."
The key addition is an explicit decision rule: "Use this when..." applied consistently across all tools that might overlap. We write tool descriptions collaboratively with the prompt engineers who write the system prompt, and we treat the decision rules as the most important part.
Pattern 5: Execution Decision Guardrails
The model deciding not to call a tool when it should is harder to catch at validation time, because there is nothing structurally wrong with the response. The fix is to make the system prompt explicit about when tool use is mandatory.
Instead of relying on the model's judgment, enumerate the conditions:
You have access to tools that retrieve live data from internal systems.
ALWAYS use a tool when:
- The user asks about a specific Jira ticket, PR, deployment, or metric
- The user asks about the current state of any system
- The user asks "what happened" or "why did X occur" about production events
- The user mentions a specific date, time range, or incident
NEVER answer from general knowledge when the user is asking about:
- Internal infrastructure, services, or team ownership
- Specific incidents or outages
- Current metric values or SLOs
If you are unsure whether to call a tool, call one. It is better to retrieve
and find nothing than to hallucinate an answer about internal systems.
The last instruction ("if unsure, call a tool") meaningfully reduced the false-negative rate in our testing. Models tend to be conservative about tool use when uncertain; this shifts the default toward action, which is the right bias for internal knowledge retrieval.
Pattern 6: Observability Per Tool Call
Tool calls should be logged with the same granularity as any other production call path: which tool was called, what arguments were generated, whether validation passed, whether execution succeeded, and what the result was. This is the minimum to debug reliability issues in production.
import time
from dataclasses import dataclass, field
@dataclass
class ToolCallRecord:
tool_name: str
raw_arguments: dict
validated: bool
validation_error: str | None
execution_success: bool
execution_error: str | None
latency_ms: float
retry_count: int = 0
trace_id: str = ""
async def instrumented_tool_call(
tool_name: str,
raw_args: dict,
trace_id: str,
) -> tuple[object, ToolCallRecord]:
record = ToolCallRecord(
tool_name=tool_name,
raw_arguments=raw_args,
validated=False,
validation_error=None,
execution_success=False,
execution_error=None,
latency_ms=0.0,
trace_id=trace_id,
)
start = time.monotonic()
try:
validated = validate_tool_call(tool_name, raw_args)
record.validated = True
result = await execute_tool(tool_name, validated)
record.execution_success = True
return result, record
except ValueError as e:
record.validation_error = str(e)
raise
except Exception as e:
record.execution_error = str(e)
raise
finally:
record.latency_ms = (time.monotonic() - start) * 1000
emit_metric(record)
The two metrics worth surfacing on a dashboard are validation pass rate per tool and execution success rate per tool. If either drops, you want to know immediately.
When Reliability Patterns Are Not Enough
There is a class of tool-calling failures that schema constraints and retries will not fix: cases where the model fundamentally misunderstands what a tool does. We had a get_deployment_history tool that accepted a service_name parameter. We measured a consistent single-digit percentage of queries where the model passed team names rather than service names, such as "payments team" instead of "payments-api". The schema accepted any string, and the values were semantically reasonable; we just needed something different.
The fix was not a schema change. It was adding a static lookup table: a prompt snippet that listed our canonical service names, updated weekly. The model matched against the list instead of inferring.
If a tool consistently receives wrong inputs despite schema constraints, the problem is usually one of these:
- The user's vocabulary does not match your tool's expected vocabulary (canonical names vs. common names)
- The tool requires context that is not in the conversation (you need the service name but the user only said "the API")
- The tool is doing too much and should be two tools
The third case is worth examining carefully. A tool that does two related things will confuse the model about when to call it. The tool calling is showing you a design problem.
Production Checklist
The patterns above compress into a checklist we use before shipping any tool-enabled system:
- [ ] Every parameter has explicit format instructions in its
description - [ ] Enum parameters use
enumin the schema, not free-form strings - [ ] Date/time parameters have format patterns in the schema and description
- [ ] Tool descriptions include "Use this when..." decision rules for any tool that might overlap with another
- [ ] A Pydantic (or equivalent) validator exists for every tool
- [ ] Validation failures feed back to the model with the error message as a tool result
- [ ] System prompt explicitly specifies when tool use is mandatory
- [ ] Tool call attempts, validation results, and execution results are logged per call
- [ ] A retry limit is enforced (we use 2 retries per tool call)
- [ ] A canonical vocabulary document exists for tools that reference internal names
A tool-calling system that skips these is reliable in demos and fragile in production. The work is not glamorous, but the retry rates and validation pass rates in your logs will tell you exactly which tools need attention and why.
Conclusion
Tool calling reliability is not a property of the underlying model. It is a property of how you define the tools, validate the calls, and handle failures. In our experience, models that misfire on a significant share of tool calls misfire on a small fraction of that after schema tightening, explicit decision rules, and validation with retry. That remaining fraction is usually a tool design problem, not a model problem.
The investment in a production tool-calling stack (schemas, validators, retry logic, observability) is roughly the same as the investment in any other production API integration. Treat it like one.
Get the next one
I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: what is the most surprising tool-calling failure you have hit in production? Reply to the email or comment below; the best one becomes the next post.
Sources
- Anthropic tool use documentation and best practices — https://docs.anthropic.com/en/docs/build-with-claude/tool-use
- OpenAI function calling reliability patterns — https://platform.openai.com/docs/guides/function-calling
- Pydantic validation library documentation — https://docs.pydantic.dev/latest/
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment