Scoring Tool-Call Accuracy, Not Just Final Answers

A fixture is a recorded run with enough detail to replay and score it. Resist the urge to store only the final answer. The useful unit is the invocation, and the second useful unit is the task, which groups invocations into a run.

{
  "task_id": "refund-partial-014",
  "input": "Customer says order 4821 arrived damaged and wants a partial refund.",
  "allowed_tools": ["lookup_order", "check_refund_policy", "issue_refund", "send_email"],
  "expected_invocations": [
    {"turn": 1, "tool": "lookup_order", "args": {"order_id": "4821"}},
    {"turn": 2, "tool": "check_refund_policy", "args": {"order_id": "4821", "reason": "damaged"}},
    {"turn": 3, "tool": "issue_refund", "args": {"order_id": "4821", "amount": 24.50}},
    {"turn": 4, "tool": "send_email", "args": {"template": "refund_confirmation", "order_id": "4821"}}
  ],
  "actual_invocations": [
    {"turn": 1, "tool": "lookup_order", "args": {"order_id": "4821"}, "result": {"status": "delivered", "total": 49.00}},
    {"turn": 2, "tool": "issue_refund", "args": {"order_id": "4821", "amount": 49.00}, "result": {"ok": true}},
    {"turn": 3, "tool": "send_email", "args": {"template": "refund_confirmation", "order_id": "4821"}, "result": {"ok": true}}
  ],
  "final_answer": "I've processed your refund."
}

Four fields do the work. Tool name supports routing scores. Arguments support value scores. Result captures what the environment returned, which matters for detecting ignored errors and for making the fixture replayable. Turn index supports order scores and lets you compare two runs of the same task position by position.

Two design choices save pain later. First, keep expected_invocations as a sequence of steps with alternatives where multiple valid paths exist, rather than one rigid script; τ²-bench grades the final database state rather than individual calls precisely because several call paths can reach the same goal per Splunk's summary of the benchmark. Second, store the result your mock tool returned, not the result the real system would return, so replay is deterministic.

If your harness already validates tool arguments with JSON Schema, reuse that machinery for fixtures. The Python jsonschema library raises a ValidationError whose validator field names the failed keyword and whose path and json_path locate the failing value documented in the jsonschema error guide. That gives you free attribution at field level: an argument failure on amount with validator type is a different bug from one with validator minimum.

Score three dimensions per task, then roll up.

Tool selection score. For each expected step, check whether the recorded tool name matches any allowed name for that step. Count matches over matched steps.

Argument score. For steps where the tool name matches, compare arguments field by field. Prefer exact equality on scalar fields and set equality on arrays. For free-text fields, define an explicit normalizer (trim, lowercase, strip punctuation) and version it alongside the fixture, because a silently changed normalizer shows up as a phantom regression.

Order score. Compute the longest common subsequence between expected and actual tool-name sequences. Divide by expected length. This tolerates a missing step without cascading every later step into a failure, which a strict positional comparison does not.

Then attribute the task to one failure mode. Use a fixed precedence so two engineers scoring the same run reach the same verdict:

The precedence rule: if any step has a wrong tool, the task is a wrong-tool failure, even if it also has argument errors, because the argument comparison was never meaningful. If tool names all resolve but order diverges, mark it wrong-order before scoring arguments as a secondary metric. Record the secondary metrics anyway; they tell you whether fixing routing will leave you with an argument problem underneath.

Here is the comparator in a compact form. It is deliberately boring code, because an evaluation harness that is hard to read will not be maintained.

def score_task(expected, actual, allowed_by_turn):
    name_hits = sum(
        1 for e, a in zip(expected, actual)
        if a["tool"] in allowed_by_turn.get(e["turn"], {e["tool"]})
    )
    arg_errors = []
    for e, a in zip(expected, actual):
        if a["tool"] != e["tool"]:
            continue
        for key, want in e["args"].items():
            got = a["args"].get(key)
            if got != want:
                arg_errors.append({"turn": e["turn"], "field": key, "want": want, "got": got})
    order_score = lcs_len([e["tool"] for e in expected], [a["tool"] for a in actual]) / len(expected)
    mode = classify(name_hits, arg_errors, order_score, len(expected), len(actual))
    return {"tool_score": name_hits / len(expected), "arg_errors": arg_errors,
            "order_score": order_score, "failure_mode": mode}

Note the trap: zip silently truncates to the shorter list, so a run with four expected steps and two actual steps scores a perfect tool name rate on the two it attempted. Handle length mismatch before scoring, and count the missing steps explicitly. This is the single most common way a homemade harness flatters a model.

One attempt per task is not a reliability measurement. Splunk's evaluation piece points out that public benchmark scores often overstate reliability because they rely on single-attempt success, and argues for pass^k style consistency metrics as a better predictor of production behavior in the same tool-calling article. The practical version: run each task k times, report the fraction of tasks that passed all k runs, and report the per-model failure-mode mix across all runs.

A worked example with numbers, all illustrative. Take 120 fixtures, run each 5 times, so 600 runs per model, and score them with the comparator above.

Read the table as a decision rule, not a leaderboard. Model A and Model B are within three points on the headline number, but they need opposite fixes. Model A's dominant mode is wrong arguments, so the work is schema and examples: tighten field types, add enums for template, put one corrected example per tricky tool in the prompt. Model B's dominant mode is wrong tool, so the work is routing: shorten and disambiguate tool descriptions, remove tools that overlap, and check that the tool list itself is not the problem. Model C needs planning help, which is usually a prompt and loop-structure change, not a schema change.

For scale, the cost differences between models on a full function-calling benchmark run can be large: Splunk reports a full BFCL run costing $86.55 on one model and $298.47 on another, within eight points of each other on score in the same article. If you are running 600 runs per candidate model against a live API, prototype with fixtures and mocks, then confirm on a small live sample. Pin one API layer across models so the cost comparison is real; Quanta Inference gives you text and tool-capable models behind one OpenAI-compatible key, prepaid and priced in dollars, at https://quantaintelligence.ai/inference.

A plain retry is the same prompt at the same temperature, roughly the same odds, as one practitioner write-up on retry loops puts it in this validation retry pattern. Feed the validation error and the model's own bad output back into the next prompt and ask it to fix that named thing. The write-up notes the tradeoff: it costs an extra call on a bad response and only works when the bad output is parseable enough to serialize back into the prompt, so cap attempts.

Apply the same idea to tool calls, with one addition: quote the schema error path, not just the message. The jsonschema error object carries validator, validator_value and json_path, so you can say "field amount failed type: expected number, got string" instead of pasting a wall of prose per the jsonschema error documentation. Models patch named, concrete mistakes better than opaque ones.

Watch for validator drift. A schema that passed for weeks can start failing without any change on your side, as one team documented when their structured output schema began failing validation overnight in this OpenAI community report. Keep the schema in version control, run a tiny canary fixture set on every deploy of your harness, and treat a sudden jump in argument failures as a possible validator or API change before you blame the model. The community thread is a user report, not an official changelog, so the operational takeaway is detection and versioning, not a claim about what changed.

Build in this order. Each item has a failure it prevents.

1. Write ten fixtures by hand from real traces. Not synthetic prompts. Real traces carry the ambiguity that synthetic ones miss. Target the tools where support tickets already come from.
2. Freeze the mock tools. Every fixture records the result each mock returned, so replays are deterministic and a score change means a model or prompt change.
3. Version the normalizers. Argument comparison rules belong in code with a version string; log the version in every report row.
4. Handle length mismatch explicitly. Assert on the number of invocations before comparing position by position.
5. Attribute one mode per task. Use the precedence table above and write down the rule; inconsistent attribution makes the per-model table meaningless.
6. Run k repeats. Report all-k pass rate, not the average of single attempts.
7. Gate on the failure mode that matters for your product. If a wrong refund amount is a compliance issue, set the gate on argument score alone and accept a lower overall pass rate.

Two smaller checks that catch real bugs. First, validate tool arguments against their JSON Schema at record time and store the validation result in the fixture; a missing required field is then visible without re-running the model. Second, log token counts per task. An agent that passes but calls twelve tools where four were expected is wasting money, and the extra-call column in your table is where that shows up.

Treating agent behavior as schema compliance plus trace scoring changes what you compare. A function-calling benchmark measures individual calls, and call accuracy is necessary but not sufficient because a valid call still fails if the checks or updates it was supposed to trigger were skipped per NVIDIA's evaluation post. Your own fixtures close that gap for your toolset, because they encode the dependencies your domain cares about. If you would rather hand the harness build and the failure analysis to someone who does it for a living, a specialist can build it with you at https://quantaintelligence.ai/.