Evaluating tool-call correctness: right tool, right arguments, result used correctly

01Four layers, four failures

user request
   │
   ▼
0. decide      call a tool, or answer directly?         wrong → unnecessary call, or missing call
   │
   ▼
1. select      which tool?                              wrong → right intent, wrong tool
   │
   ▼
2. fill        which arguments?                         wrong → right tool, wrong or guessed values
   │
   ▼
3. use         what to do with the result?              wrong → ignored, misread, or hallucinated past it

Each layer can be right while the next is wrong, and the fixes differ: layer 0 and 1 failures are about tool descriptions and the system prompt; layer 2 failures are about parameter descriptions, examples and what the model was told about defaults; layer 3 failures are about how results are formatted and what the prompt says about using them. A single "tool accuracy" number mixes all four.

02Layer 0: whether to call

Before any tool is chosen, the model decides whether one is needed. Evaluation sets often omit the cases where the right answer is "no tool", and then the model's tendency to call tools it does not need goes unmeasured. Include cases where the request is answerable directly, where it is out of scope, and where information is missing and the right move is to ask. Score with a two-by-two: called when it should, did not call when it should not, and the two error types, which have different costs. An unnecessary call wastes a step; a missing call produces an answer from memory that should have been looked up.

03Layer 1: which tool

With a tool set of more than a few, selection is a classification problem and should be reported as one: a confusion matrix with one row per tool plus a row for "no call". Per-tool precision and recall then show which tools are over-chosen and which are under-chosen, and the off-diagonal cells show which pairs the model confuses. A pair that is confused often is a pair whose descriptions do not draw the boundary, which the schema note covers. An aggregate accuracy hides all of this; the matrix is the useful object.

04Layer 2: which arguments, and why exact match is wrong

Given the right tool, were the arguments right? The tempting method is to compare the model's JSON to the reference JSON. Consider what that gets wrong:

Six of the seven "wrong" verdicts are false, and the one true verdict is scored the same as the six false ones. Exact match both understates argument accuracy and fails to distinguish a harmless formatting difference from sending the email to the wrong person.

05Schema-aware comparison, type by type

The schema already says what each field is. Use it to choose the comparison.

Two more rules make this usable.

• Weight by consequence. Mark each field in the schema as critical or not. A wrong recipient, amount, or target id is a failed call regardless of the other fields; a wrong optional note is a minor deduction. Report the critical-field failure rate separately, because it is the one that corresponds to harm.
• Distinguish wrong from guessed. If an argument's value does not appear in the user's request or any prior tool result, it was invented, even if it happens to match the reference. Flag unsourced values as a separate category; they are the arguments that will be wrong next time.

The output of the comparison is a per-field verdict, which aggregates to a per-call score and, across the set, to a per-field error rate per tool. That last table is the actionable one: it says which parameter description to rewrite.

06Layer 3: how the result was used

The layer most evaluations skip. The call was right, the result came back, and then what? Four things can go wrong: the model ignores the result and answers from memory; it misreads it, reporting a status the result does not say; it treats an error as a success or an empty result as a completion; or it uses a stale result from an earlier step.

You cannot score this by looking at the response alone, because a response that happens to match the truth may have been produced without reading the result. The method is counterfactual: change what the stub returns and check that the response changes to match.

case: "what is the status of order 98765?"
  stub returns  {status: "shipped"}          → response must say shipped
  stub returns  {status: "cancelled"}        → response must say cancelled
  stub returns  {error: "not found"}         → response must say it could not find the order, not invent a status
  stub returns  []                           → response must say no results, not "done"

score: fraction of variants where the response tracks the result

A model that answers "shipped" in all four variants scored 100 percent on tool selection and arguments and is not using the tool at all. This test finds that in one run, and it is cheap, because the variants share the request and differ only in the stub.

Faithfulness judging complements it: for each claim in the response, is it supported by a tool result in the trace? Claims about the world with no supporting result are the hallucinated-past-the-tool failure.

07Multi-step sequences

When a task takes several calls, two more questions arise. Was the sequence acceptable, which the success-versus-process note covers with required calls, ordering constraints and budgets. And were the derived arguments right: when call two's argument should come from call one's result, did it?

Score derivation explicitly. For each argument in a later call that the task says depends on an earlier result, check that its value equals the right field of that result. A model that looks up the order, receives customer_id: C-4471, and then calls the refund tool with customer_id: C-4471 derived correctly; one that calls it with a customer id from the user's message, or from nowhere, did not, even if the value happens to match. The counterfactual method works here too: change the first stub's returned id and check the second call follows it.

08Pitfalls

• One reference call when several are valid. If "search the knowledge base" and "fetch the policy document" both answer the question, a reference that lists one penalises the other. Mark acceptable alternatives in the case.
• Reference arguments that are themselves one of several right answers. A search query has many equivalent forms; store the intent and judge, do not store one string and match.
• Scoring against live services. Results change, so the reference drifts. Stub the tools with fixed state per case, which also makes the counterfactual tests possible.
• Ignoring the no-call cases. Without them, over-calling is invisible.
• Reporting one number. Decision rate, selection matrix, per-field argument error rates with critical fields separate, result-use tracking rate. Four things, and each points at a different fix.

09Takeaways

1. Four layers: whether to call, which tool, which arguments, how the result was used. Score each.
2. Include no-call cases, or over-calling goes unmeasured.
3. Selection is a confusion matrix, and the off-diagonal cells name the descriptions to fix.
4. Exact-match arguments are wrong for most fields. Compare by schema type, weight by consequence, flag unsourced values.
5. Test result use counterfactually. Change the stub, check the response follows.
6. In sequences, check that derived arguments were actually derived, not coincidentally correct.