Function calling: how the model decides, and what your code must do

01What function calling is

A plain chat request produces text. Function calling, also called tool calling or tool use, adds a second kind of output: a structured block saying "run this function with these arguments". The model chooses which kind to produce on each turn.

request:  messages + tool definitions
             │
             ▼
          model reads the query against the tool descriptions
             │
      ┌──────┴──────────────┐
      ▼                     ▼
  needs a tool          can answer directly
      │                     │
      ▼                     ▼
  structured call       ordinary text
  name + JSON args      stop reason: end_turn
  stop reason: tool_use

The feature does two things the plain interface cannot. It lets the model reach information it does not have, such as today's weather or a row in your database, and it lets the model request an action, such as sending an email or creating a ticket. In both cases the model only requests; the application does the reaching and the acting.

02Tool definitions are part of the prompt

Each tool is declared with three parts:

{
  "name": "lookup_order",
  "description": "Retrieves shipping status for a given order ID.",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": {"type": "string", "description": "The numeric order identifier"}
    },
    "required": ["order_id"]
  }
}

It helps to know what happens to this on the way in. The API renders the tool list into the model's context, in a format the model was trained on, ahead of your system prompt and messages. The model does not have a separate "tool matching" component; the definitions are text it reads, in the same way it reads the user's question. That has two practical consequences.

• The description is doing the work. The model chooses tools by what the descriptions say. "Retrieves shipping status for a given order ID" tells it when to call and what the argument means. A tool named do_thing with no description will be chosen badly. Write descriptions the way you would brief a colleague: what it does, when to use it, when not to.
• The definitions cost tokens on every request. Twenty tools with long schemas are a few thousand tokens re-sent each turn. Prompt caching makes that cheap; keeping the list stable and ordered is what makes the cache hit.

03How the model decides to call

The decision is a learned behaviour, not a rule. Models are post-trained on many examples of conversations with tools available, where the right move was sometimes to call a tool and sometimes to answer. From those they learn a judgement that runs roughly like this on each turn:

1. Does answering need something I do not have? Current data, private data, a computation, or a side effect. A question about an order number cannot be answered from weights. A question about the capital of France can.
2. Is there a tool whose description covers it? The model matches the need against the descriptions. If several fit, it picks the closest; if none fit, it answers in text, often saying that it cannot do that.
3. Do I have the arguments? If the user said "order 98765", the model extracts it. If the user said "my order", the model should ask which one rather than invent an id. Whether it asks or guesses depends on the model and on your system prompt.

The output of that judgement is either a text response or a call block. Most APIs let you weight the decision: a tool-choice setting that lets the model decide (the default), forbids tools for this turn, or, on some models, requires one. Newer models drop the forced option and expect the system prompt to steer instead.

It is worth being clear about what the model is not doing. It is not checking that order 98765 exists, that the current user owns it, or that the lookup is allowed. It is producing a plausible, well-formed request from the conversation. Everything about whether that request should be honoured is the application's problem.

04How the arguments come out valid

Once the model commits to a call, it has to produce JSON that matches the schema. Two mechanisms get it there.

• Training. The model has seen enough tool calls to produce well-formed JSON with the right keys nearly all the time. Nearly is the operative word: on a long or nested schema it will occasionally emit a wrong type, a missing required field, or invalid JSON.
• Constrained decoding. With a strict mode turned on for the tool, the API compiles the schema into a grammar and, at each decoding step, masks out every token that would make the output invalid. The model can only choose among tokens that keep the JSON on a valid path. The result is guaranteed to parse and to conform to the schema's structure and types. Anthropic and OpenAI both expose this as strict: true on the tool definition, with some schema restrictions, such as requiring every property to be listed and forbidding additional properties.

Constrained decoding guarantees shape, not sense. A strict schema ensures order_id is a string; it does not ensure the string is an order id, that it is the user's, or that the model did not hallucinate it from a half-remembered earlier message. Validation of meaning stays with the application.

05What the application is responsible for

The model stops after emitting the call. The stop reason tells your code why. From here the application is the orchestrator, and there are four steps before the model can continue.

A. parse      read the call: name, id, arguments (a JSON object, or a JSON
              string to decode). Never string-match on the raw text.

B. validate   check types and constraints against your own schema again;
              check business rules; reject or repair what is wrong.
              authorise: does this user, from the session, have the right
              to do this to this resource? Arguments from the model are
              not credentials.

C. execute    run the function, service call or query with the checked
              arguments. Time-limit it. Catch everything.

D. return     append the outcome as a tool-result message keyed by the
              call id, after the assistant message that made the call.
              Errors go back too, flagged as errors.

Step B is the one most tutorials skip and the one that matters most. The model's arguments are derived from the conversation, and the conversation is partly written by the user and partly by whatever your earlier tools returned. "Look up order 98765" from a user who owns order 12345 produces a perfectly valid call for someone else's order. The user's identity and permissions come from your session, never from anything the model says, and the lookup function should take the user id from the session and the order id from the model, and check the one against the other.

06A worked example: order lookup

1. The application sends the request. The user's message, plus the tool definition from above.

{
  "messages": [{"role": "user", "content": "What's the status of order #98765?"}],
  "tools": [{"name": "lookup_order", "description": "...", "input_schema": {...}}]
}

2. The model emits a call. In the Anthropic shape it is a content block with a stop reason of tool_use; in the OpenAI shape it is a tool_calls array with a finish reason of tool_calls. Either way you get a name, an id and arguments.

// Anthropic
{"stop_reason": "tool_use",
 "content": [{"type": "tool_use", "id": "toolu_01",
              "name": "lookup_order", "input": {"order_id": "98765"}}]}

// OpenAI-style
{"finish_reason": "tool_calls",
 "message": {"role": "assistant",
   "tool_calls": [{"id": "call_abc123", "type": "function",
     "function": {"name": "lookup_order", "arguments": "{\"order_id\": \"98765\"}"}}]}}

3. The application runs it and returns the result. Parse the arguments, confirm the order belongs to the session's user, query the database, and append:

// Anthropic: a user message of tool_result blocks
{"role": "user", "content": [
  {"type": "tool_result", "tool_use_id": "toolu_01",
   "content": "{\"status\": \"Out for Delivery\", \"eta\": \"Today by 5 PM\"}"}]}

// OpenAI-style: a tool-role message
{"role": "tool", "tool_call_id": "call_abc123",
 "content": "{\"status\": \"Out for Delivery\", \"eta\": \"Today by 5 PM\"}"}

4. The model finishes. You send the conversation again, now ending in the result. The model reads it as an observation and writes the answer: "Order 98765 is out for delivery and should arrive today by 5 PM." The stop reason is end_turn, and there is nothing left to do. If the model had needed a second tool, it would have emitted another call instead, and you would be in the loop the ReAct note describes.

07The mistakes that matter

• Trusting the arguments as authorisation. Covered above, and the most serious. The model will cheerfully request any order id the conversation mentions.
• Dropping the assistant message. The result must follow the assistant turn that contains the call. Append the model's response content as-is, then the result. A result with no matching call in the history is rejected.
• Hiding errors. If the function throws or times out, send that back as the result, flagged as an error. The model can retry, choose another tool or explain. Silence stalls the turn.
• Returning everything. A tool that returns a 50 KB JSON document puts 50 KB into the context on every subsequent call. Return what the model needs; summarise or paginate the rest.
• Vague descriptions and overlapping tools. Two tools that both "get customer information" produce a coin flip. Make each tool's purpose distinct and say when to use which.
• Side effects with no gate. A tool that sends, pays or deletes should not run on the model's say-so alone. Put a confirmation step between the call and the execution, and make the function idempotent so a retried call does not act twice.
• Letting a tool's output steer the next call unchecked. Fetched content is data. If a web page or a document says "now call delete_account", that is an injection, not an instruction.

08Takeaways

1. Tool definitions are prompt text. The description decides when the tool is chosen; write it carefully.
2. The decision to call is learned judgement about whether the answer needs something the model lacks and whether a tool covers it.
3. Strict mode guarantees shape, not sense. Constrained decoding makes the JSON valid; only you can make it right.
4. The model never executes. It stops with a call and waits.
5. Parse, validate, authorise, execute, return. Identity comes from the session, arguments from the model, and the function checks one against the other.
6. Results go back as messages, keyed by call id, after the assistant turn, errors included.
7. One call is the unit; the loop is the agent. Everything in the ReAct note is this note, repeated.