Chapter 4: Tool Use & Function Calling
How agents reach out of the LLM and into the world. Function calling fundamentals, JSON-mode and tool schemas, parallel tool calls, the Model Context Protocol (MCP) for portable tools, and the discipline of handling tool results and errors gracefully.
- Topics
- 5
- Videos
- 3
- Min read
- 17
Chapter Overview
An LLM that cannot call tools is a calculator that cannot add. Tool use — also called function calling — is the API by which an agent asks the controller to execute a function and feed the result back into the context. It is the single most important capability the major LLM providers added in 2023, and it is what turns a chatbot into an agent.
Before function calling, agents emitted tool calls as raw text (Action: search['query']) and the controller parsed the string. Brittle: every escape character was a bug, every model update changed the format. Function calling moves the parsing into the inference layer: the model is trained to emit a structured tool-use block, the API guarantees valid JSON matching your schema, and the controller just dispatches the call.
This chapter covers:
- Function calling fundamentals — the JSON-schema contract between model and controller
- Tool schemas & JSON mode — designing schemas the model can follow reliably
- Parallel tool calls — fan out independent tools, wait on the slowest
- MCP — the Model Context Protocol for portable, multi-vendor tool servers
- Tool result handling & errors — what to put in the observation, how to surface failures
Chapter Roadmap
Click any topic to jump in
Function Calling
The schema-validated contract between LLM and controller. The single most important agent capability.
What the model emits, and how many it emits at once
Schemas & JSON Mode
How to design schemas the model picks reliably. JSON mode vs function calling vs structured output.
Parallel Tool Calls
Fan out independent tools, wait on the slowest. 4× wall-clock speedup is typical.
MCP
The portable tool protocol — write tools once, run them in any MCP-aware agent.
Result Handling
Be terse, be honest about errors, be structured. Observations are the binding context constraint.
Topics
5 topics in this chapter
Chapters 1 through 3 gave the agent a loop, plans, and the ability to learn from failure — but everything still happened inside the model's own text. An agent that cannot call tools is a calculator that cannot add: it can reason about the world but never touch it. Function calling is the mechanism that reaches out of the LLM and into real systems, and it is the single most important capability the major providers shipped in 2023. This topic establishes the contract precisely. You describe your tools to the model as JSON schemas; the model decides when to call one and emits a structured tool-use block; the API guarantees that block is valid JSON matching your schema; and the controller simply dispatches it. That is a decisive upgrade over the brittle old world of parsing Action: search['query'] out of raw text, where every escape character was a bug and every model update broke the format. We trace the three-step request/response/result loop and the schema-design rules that decide whether the model reaches for the right tool at all.
Definition
Function calling is the schema-validated contract by which an agent exposes tools to an LLM: each tool is described by a name, a description, and a JSON schema for its parameters. The model responds with either a final text answer or a structured tool-use block whose arguments the API guarantees conform to the schema, which the controller then executes.
In this topic
- 1The Three-Step Loop
- 2Tool Schemas — the Contract
The Three-Step Loop
Every function-calling cycle has the same shape:
- You send a request to the LLM with the user message + a list of tool schemas (each: name, description, JSON-schema for parameters).
- The LLM responds with EITHER (a) a regular text completion (it is done) OR (b) a tool-use block containing the chosen tool name and a JSON object matching the schema for that tool's parameters.
- You execute the tool, then send a follow-up request with the original messages + the tool-use block + a tool-result block containing whatever the function returned.
The LLM then either calls another tool or produces a final text answer. The loop repeats until you get text instead of a tool call.
In the Anthropic API: messages alternate user / assistant and tool results come as user messages with a tool_result content block. In the OpenAI API: tool results come as tool role messages. Same shape, different wire format.
Tool Schemas — the Contract
A tool schema looks like this (Anthropic / OpenAI use the same JSON-Schema dialect):
{
"name": "get_weather",
"description": "Fetch current weather for a city.",
"input_schema": {
"type": "object",
"properties": {
"city": { "type": "string", "description": "City name (e.g. 'San Francisco')" },
"units": { "type": "string", "enum": ["metric", "imperial"], "default": "metric" }
},
"required": ["city"]
}
}
Four rules for schemas the model can use reliably:
- Description is the prompt. The model picks tools based on description text. Be specific: 'Search the customer support knowledge base. Use ONLY for support questions, NOT general web search.'
- Parameter descriptions matter as much as the tool description. The model will hallucinate parameter values if it doesn't know what they mean.
- Use enums where possible. "status": {"enum": ["open", "closed"]} constrains the model to valid values.
- Required vs optional. Mark optional anything with a sensible default. The fewer required parameters, the less chance the model picks one tool because another's required field is intimidating.
An agent has 5 tools but the model never calls one of them. What's likely wrong with the schema?
The previous topic treated function calling as the way to get action out of a model, but it is really one instance of a more general problem: coaxing the model to emit structured output you can parse reliably. This topic widens the lens to the full toolbox and, just as importantly, teaches the schema-design discipline that determines whether any of it works. Alongside function calling sit JSON mode, which forces the response to be parseable JSON with no schema enforcement, and structured outputs, which constrain the response to an arbitrary schema even when there is no 'tool' to call — with grammar-constrained decoding underneath both. Choosing among them is a real decision: use function calling when there is an action to take, structured output when you only need a typed response, and avoid bare JSON mode, which gives parsing without validation. We also cover why schemas are themselves prompt engineering — granularity, parameter ordering, sensible defaults — because the most common production failure here is not a broken parser but a model that picks the wrong tool from a menu that is too large or too vague.
Definition
Structured output is the general problem of making an LLM emit machine-parseable text, addressed by three related features: JSON mode forces valid but unconstrained JSON, function calling binds output to a chosen tool's schema, and structured-output modes constrain a response to an arbitrary schema. Schema design — granularity, ordering, defaults — is itself a form of prompt engineering.
In this topic
- 1Schema Design Discipline
- 2JSON Mode vs Function Calling vs Structured Output
Schema Design Discipline
Schemas are also a form of prompt engineering. Three high-impact decisions:
- Granularity. One large tool with many parameters vs. several small tools. The model is better at picking between distinct tools than at filling many parameters of one tool. Prefer many narrow tools.
- Parameter ordering. The model emits parameters in schema order. Put the most important parameter first so the model 'commits' to its plan before filling in details.
- Default values. Every default reduces the model's cognitive load. If 95% of search calls use limit=10, make it the default and let the model omit it.
A common mistake: importing all 80 endpoints from your REST API as 80 tools. The model picks badly when the menu is too large. Group by use case (search_*, mutate_*, notify_*) and have a router-LLM pre-pick a category, OR use a hierarchical structure (agent_action(category=X, ...)).
JSON Mode vs Function Calling vs Structured Output
The provider APIs offer three related features — easy to confuse:
- JSON mode (OpenAI, others): forces the model to emit some valid JSON. No schema enforcement; the JSON shape is up to the model.
- Function calling / Tool use (Anthropic, OpenAI): the model picks a tool from your list AND emits a JSON object matching that tool's schema. Schema-enforced.
- Structured output (OpenAI's response_format=json_schema, Pydantic-based libraries): force the model's response to match an arbitrary JSON schema, even when there is no 'tool' concept. Schema-enforced.
Use function calling when there's a real action to take. Use structured output when you just want a parseable response (extraction, classification with metadata, etc.). Avoid raw JSON mode unless you really need free-form JSON — it gives you parsing reliability without schema validation, which is the worst of both worlds.
You want the model to extract {name, email, requested_action} from a customer message. Function calling, JSON mode, or structured output?
So far the agent has called one tool at a time, waited for the result, and only then decided what to do next. That sequencing is exactly right when each step depends on the last, but it is pure waste when the steps are independent: asking a plain ReAct loop to 'compare the weather and exchange rates in Paris, London, and Tokyo' fires six tool calls one after another, and wall-clock time grows linearly with the count. This topic introduces parallel tool calls, where the model emits several tool-use blocks in a single response, the controller dispatches them concurrently, and the agent waits only on the slowest — turning a sum of latencies into a maximum, often a 4× speedup. We walk the concrete wire shape in the provider APIs (multiple tool_use blocks out, all tool_result blocks back in one message) and, crucially, the judgement of when to use it. Parallelism helps only when calls are genuinely independent; it hurts when calls have side effects that must be sequenced, share a rate limit, or when the model's plan should adapt to each result. The model makes the call, but a good system prompt is what makes it choose well.
Definition
Parallel tool calls let the model emit multiple tool-use blocks in a single response, which the controller executes concurrently and returns together in one message. Wall-clock time for the batch becomes the maximum of the individual latencies rather than their sum. It is safe only when the calls are independent — no shared side effects, rate limits, or data dependencies.
In this topic
- 1How Parallel Calls Work
- 2When Parallelism Helps and Hurts
How Parallel Calls Work
The provider APIs support multiple tool_use blocks in a single assistant response. Concrete shape (Anthropic):
{
"role": "assistant",
"content": [
{ "type": "text", "text": "I'll fetch all three at once." },
{ "type": "tool_use", "id": "a", "name": "weather", "input": {"city": "Paris"} },
{ "type": "tool_use", "id": "b", "name": "weather", "input": {"city": "London"} },
{ "type": "tool_use", "id": "c", "name": "weather", "input": {"city": "Tokyo"} }
]
}
The controller dispatches all three concurrently, awaits a Promise.all (or equivalent), and sends all three results back in a single user message:
{ "role": "user", "content": [
{ "type": "tool_result", "tool_use_id": "a", "content": "..." },
{ "type": "tool_result", "tool_use_id": "b", "content": "..." },
{ "type": "tool_result", "tool_use_id": "c", "content": "..." }
] }
The wall-clock for the three tool calls becomes instead of their sum. For 4 calls of 800 ms each: 800 ms instead of 3200 ms — a 4× speedup.
When Parallelism Helps and Hurts
Helps when: tool calls are independent — the result of one is not needed to construct the input of another. Weather lookups for different cities, RAG retrievals against different indexes, multi-region searches, batch enrichment of a list of records.
Hurts when:
- Tool calls have side effects that should be sequenced (update_record then audit_log)
- Tool calls are rate-limited by the same upstream — you'll get 429s firing them concurrently
- The model's plan is adaptive — if the result of one call should determine whether to make another, parallel forces a worse plan
The model decides. With the right system prompt ('use parallel tool calls when the tools are independent'), modern Claude/GPT models pick well. Without it, they default to sequential out of caution.
Once you have written a handful of tools, a scaling problem appears: every model provider has slightly different tool schemas, every framework has its own tool registry, and wiring up each new data source means writing yet another bespoke adapter. Model Context Protocol (MCP) is Anthropic's open standard — released late 2024 — for solving that with a single portable wire protocol. This topic explains its architecture and why it matters to anyone building agents. An MCP server is a standalone process that exposes tools, read-only resources, and reusable prompts; an MCP client, embedded in your agent, discovers servers and calls their capabilities; and a JSON-RPC transport connects them over stdio, websocket, or HTTP. The payoff is that a tool server written once — GitHub, Postgres, Slack, Sentry — plugs into any MCP-aware client with no code changes, giving you vendor portability, reuse across agents, and auth handled at the protocol layer. The ecosystem is young and in flux, but MCP is the strongest candidate to become the 'USB-C of agent tooling', so building new internal tools as MCP servers costs little and buys real optionality.
Definition
The Model Context Protocol (MCP) is an open standard defining a portable wire protocol between MCP servers — processes exposing tools, resources, and prompts — and MCP clients embedded in agents, connected by a JSON-RPC transport. A tool written once as an MCP server runs in any MCP-aware client, providing vendor portability, cross-agent reuse, and protocol-level authorization.
In this topic
- 1The MCP Architecture
- 2Why MCP Matters for Agent Builders
The MCP Architecture
Three components:
- MCP server. A standalone process that exposes a set of tools, resources (read-only data), and prompts (reusable templates). Implemented in any language; runs as a subprocess, container, or remote service.
- MCP client. Embedded in the agent. Discovers servers, lists their capabilities, calls their tools. Supports multiple servers concurrently.
- MCP transport. The wire protocol (JSON-RPC over stdio, websocket, or HTTP).
The ecosystem of MCP servers is growing fast: GitHub, Postgres, Slack, Linear, Sentry, AWS, Filesystem — most have first-party or community MCP servers. Plug them into Claude Desktop and you get a desktop agent with that capability, with no code changes.
Why MCP Matters for Agent Builders
Three concrete wins:
- Vendor portability. Write your tools once as MCP servers; they work with any MCP-aware client. If you migrate from Anthropic to a different provider that supports MCP, your tools come with you.
- Reusability across agents. Your customer-support agent and your dev tools agent both need access to the user database. Both connect to the same MCP database server — no per-agent re-implementation.
- Permissions and auth at the protocol layer. MCP defines how clients authorize servers and how servers expose available capabilities. Centralized: change the policy in one place instead of in every agent.
MCP is still young (released late 2024) and the ecosystem is in flux. But it is the strongest candidate for becoming the 'USB-C of agent tooling.' Build new internal tools as MCP servers when reasonable; you cost yourself nothing and gain optionality.
You're building an agent that needs to read from a Postgres database, post to Slack, and create GitHub issues. Three tools — but each provider has a different SDK. How does MCP help?
Every topic so far has been about getting the agent to call a tool; this closing one is about what comes back, which is where production agents quietly live or die. Tools fail constantly — APIs go down, queries return nothing, JSON arrives malformed, rate limits trip — and, more subtly, tools that succeed can return so much data they choke the context window within a few steps. What you place in the observation is the only signal the agent has, and it decides whether the agent recovers or spirals into a loop. This topic distills three principles, each easy to violate under deadline pressure. Be terse: truncate, project, summarize, or cache-by-reference so a 50KB blob never lands whole in the prompt. Be honest about errors: return the real error message so the model can diagnose and retry, redacting only genuine secrets. Be structured: use one consistent result envelope so the model learns where to look. Together these treat the context window as the scarce resource it is, making tool-result design at least as important as tool-parameter design — the discipline that ties the whole chapter together.
Definition
Tool-result handling is the discipline of shaping observations returned to the agent so they aid recovery without exhausting the context window. Its three principles are terseness (truncate, project, summarize, or cache behind a handle), honesty about errors (return real, secret-redacted error messages), and structure (a consistent result envelope the model can reliably parse).
In this topic
- 1Be Terse
- 2Be Honest About Errors
- 3Be Structured
Be Terse
A 50KB JSON blob from a Postgres query will burn through your context window in a few iterations. A 200KB GraphQL response will blow it instantly. Be aggressive about what makes it back into the prompt:
- Truncate. Long lists: top- + ... (N more truncated). Long text: first 200 tokens + last 100 tokens.
- Project. A query for users likely doesn't need every column. Have the tool accept a fields parameter and project before serializing.
- Summarize. For multi-page documents, run a small, cheap summarizer over the result before injecting. Adds ~100ms but saves ~5000 tokens per result.
- Cache by reference. Store the full result on the controller side, return only a handle (result_id: 'r-1234') and the model can ask for specific fields via a follow-up tool call.
The second mode is critical for agentic search — never put 10 search results × 5KB each into a single tool result.
Be Honest About Errors
When a tool fails, return the actual error message to the model, not a sanitized 'something went wrong.' Three reasons:
- The model can recover from specific errors. '404 Not Found: customer_id=12345' tells it to verify the ID; 'Error' tells it nothing.
- The model can fix invalid arguments. A schema validation error like 'Invalid date format: expected YYYY-MM-DD, got 03/15/2024' lets the model retry with the right format on the next call.
- The model can escalate intelligently. A '500 Internal Server Error' plus a Reflexion-style retry policy is much better than a generic failure.
The one exception: error messages that contain secrets (DB connection strings, API keys, internal hostnames). Strip those, but preserve the structure: 'Database connection failed: <internal-host>:<port>' becomes 'Database connection failed: [host redacted]:[port redacted]' — keeps the meaningful information, hides the secret.
Be Structured
Always return tool results in a consistent shape. The model has to extract relevant fields on subsequent turns; if every tool returns a different blob, the model wastes tokens parsing.
A solid baseline:
{
"status": "ok" | "error",
"data": <tool-specific>,
"error": <message, only if status==error>,
"meta": { "latency_ms": 240, "cached": false }
}
The model learns to look for data for success, error for failure. Adding a meta block helps the model adapt: if it sees cached: true it knows the freshness might be lower; if it sees high latency_ms it can choose to skip a follow-up call.
Theory Exercise
Problem:
An agent calls a search tool that returns 50 results × 4KB each. After 3 such calls, the agent's context is full and it stops working. What do you change?
Hints:
- Where in the pipeline does the bloat happen?
- Does the model actually need all 50 results?
- What's the cheapest intervention?