A Test Harness for Tool Selection Accuracy

A case is a small record: an id, the user request, the environment (system prompt plus tool list), and the expected outcome. The expected outcome is either a specific tool name or an explicit non-call label such as ask_followup or refuse_no_tool. Keep the request text verbatim as a user would type it. Rewriting requests into tidy phrasing teaches the model a dialect your users do not speak.

Cover four case types, and write down the mix:

The near-miss and negative rows are where most of the value sits. A tool list where every request maps cleanly to one tool is a test that passes on day one and never tells you anything again. When2Call exists precisely because models hallucinate tools that are not in the provided list or invent missing parameters, and those behaviors need their own labelled cases (When2Call, arXiv).

How many cases? Enough that one flip moves accuracy by a number you care about. With 100 cases, one wrong selection moves accuracy by one percentage point. With 25 cases, one flip is four points, which is too coarse to distinguish a real regression from noise. A practical floor is 60 to 120 cases for a single agent, with at least 20 in the near-miss and negative categories combined. For a multi-tool agent, MLflow's guidance flags more than eight tools registered at once as a design problem rather than a capability gap, which is a useful prompt to split the tool list before you write cases (MLflow).

Write cases from real traffic where you have it. Pull a sample of production requests, label the correct tool by hand, and keep the disagreements between two human labellers as a separate "ambiguous" bucket that you fix in the tool descriptions rather than in the test. A case two engineers cannot label the same way is a tool design bug, and it will fail in production the same way.

The harness needs the agent's actual tool call, not its narrative. Run the full pipeline for each case with tools enabled, capture the first tool call the model emits, and store it next to the expected label. Confident AI's evaluation metrics list separates tool correctness from task completion and argument correctness, and the split is worth copying: whether the right tool was chosen, whether the arguments were valid, and whether the task was finished are three different failures with three different fixes (Confident AI).

A minimal Python harness looks like this. It assumes your agent exposes a function that returns the tool name it would call, or None for a non-call.

import json, collections

CASES = json.load(open("tool_cases.json"))
# [{"id": "c001", "request": "...", "expected": "get_invoice_by_id", "env": "billing_agent"}, ...]

def chosen_tool(case):
    # run the real agent pipeline with the tool list for case["env"]
    # return the first tool name, or one of the non-call labels
    result = agent.select_tool(case["request"], env=case["env"])
    return result.tool_name or result.decision  # e.g. "ask_followup"

rows = []
for case in CASES:
    chosen = chosen_tool(case)
    rows.append({"id": case["id"], "expected": case["expected"], "chosen": chosen,
                 "ok": chosen == case["expected"], "request": case["request"]})

correct = sum(r["ok"] for r in rows)
print(f"accuracy = {correct}/{len(rows)} = {correct/len(rows):.3f}")
json.dump(rows, open("selection_results.json", "w"), indent=2)

Run each case once for the gate, and run a small subset three to five times to see how often the model flips between two answers. A case that flips run to run is not noise to be averaged away; it is a description or prompt defect you can usually fix by tightening two tool descriptions that overlap.

Two practical details save pain later. First, pin the tool list per case so a new tool landing in the registry does not silently change the environment for every historical case. Second, log the full request, the tool list hash, the model identifier and the raw tool call. When a score drops, you want to know whether the model changed, the tools changed or the prompt changed, and the log answers that in seconds.

Accuracy alone hides the shape of the failure. A model can hold steady accuracy while swapping which tools it confuses with each other, and that swap matters enormously if one of the tools deletes a record. The confusion matrix of expected against chosen tools is where the useful information lives. The classic treatment of precision, recall and F1 starts from that matrix, and routing queries to the correct tool is one of the standard applications it covers (LearnWithNK).

Build the matrix with expected tools as rows and chosen tools as columns. A worked example: 80 cases, 68 correct.

Accuracy is 68/80 = 0.85. The matrix says more: refund_order is the weak row, losing 3 cases to get_invoice and 2 to ask_followup, and ask_followup is called when a real tool existed in 3 cases. A single number would have said "85%, probably fine". The row says "refund requests are being read as invoice lookups", which points straight at two overlapping tool descriptions.

Add two per-tool numbers to the report. Recall for a tool is the share of its expected cases that were correct, the diagonal cell divided by the row total. Precision for a tool is the share of times it was chosen that were right, the diagonal divided by the column total. From the table, refund_order recall is 16/21 = 0.76 and get_invoice precision is 22/27 = 0.81. Report the worst recall across tools, not just the average, because the weakest tool is the one that will page you at night.

The OpenAI developer community thread on confusion matrices for LLM responses captures a fair objection: with free-text answers there is no exact ground truth to compare. Selection sidesteps that objection, because the chosen tool is a discrete identifier, not prose (OpenAI community). That is the argument for measuring selection before you measure answer quality: it gives you a crisp, checkable metric that does not need a model to grade it.

Run the harness against the current production build and record the result. That run is your baseline: accuracy, worst per-tool recall, and the exact set of off-diagonal cells that are non-zero. A threshold pulled from the air (say, "90%") either blocks every deploy forever or waves through a regression, depending on luck. The baseline anchors it to what you actually ship.

A threshold that has held up in practice has three parts:

1. Overall accuracy at or above the baseline, or within a small tolerance you justify from the flip rate you measured on repeated runs.
2. Worst per-tool recall at or above its baseline value, so a strong tool cannot mask a collapsed one.
3. No new off-diagonal cell that was zero in the baseline. An existing confusion between two read-only tools is a known cost; a new confusion between get_invoice and refund_order is a money-moving defect.

Part three is the decision rule that a first search result will not give you. Tolerating a small accuracy drop while forbidding new confusion cells separates "slightly noisier, same risk profile" from "picks a different, more dangerous tool". Write it as a check the build runs:

def gate(results, baseline):
    acc = sum(r["ok"] for r in results) / len(results)
    cells = collections.Counter((r["expected"], r["chosen"]) for r in results if not r["ok"])
    new_cells = {c: n for c, n in cells.items() if c not in baseline["cells"]}
    fails = []
    if acc < baseline["accuracy"] - baseline["tolerance"]:
        fails.append(f"accuracy {acc:.3f} below floor")
    for tool, recall in per_tool_recall(results).items():
        floor = baseline["recall"].get(tool, 0.0)
        if recall < floor:
            fails.append(f"{tool} recall {recall:.2f} below {floor:.2f}")
    for (exp, ch), n in new_cells.items():
        fails.append(f"new confusion {exp} -> {ch} ({n} cases)")
    return fails

Run this on every pull request that touches prompts, tool schemas or descriptions, and on every model change. Microsoft's guidance recommends automated scans at design, development, pre-deployment and post-deployment with scheduled runs, and the same cadence applies here: the gate at deploy time, plus a scheduled run against production traffic samples to catch drift between releases (Microsoft Foundry). Keep the baseline file in version control next to the cases, and require a second reviewer to move it.

The failure modes in the matrix point to different fixes, and applying the wrong one wastes a week.

Anthropic's tool-writing guidance makes the description the first lever: namespacing tools to define clear functional boundaries, returning meaningful context, and prompt-engineering the descriptions and specs are listed as the core principles for high-quality tools (Anthropic engineering). When two tools share a confusion cell, the fix is almost always in the text the model reads, not in the selection layer. MLflow's practitioner guide goes further on schema design, recommending narrow, job-specific tools with typed input and output schemas, such as get_invoice_by_id(invoice_id: str) rather than a general query_database(sql: str), because the narrower tool is easier for the model to choose correctly and safer when it does (MLflow).

There is a security angle to keep in view. Zenity's least-agency argument notes that an agent can stay entirely inside its authorized permissions and still act inappropriately for its purpose, and proposes measuring the gap between what an agent can reach and what it acts on autonomously (Zenity). Selection accuracy is a direct input to that: every mismatch is an autonomous decision to touch a system you did not intend. Default-deny tool permissioning, where a new agent starts with no tools and each is added and scoped explicitly, turns a selection error into a refusal rather than an action, which is the right failure direction for a gate to protect (Skyrelis).

If you are running agents with a large tool surface, the effort of writing and maintaining these cases grows with the number of providers and environments you test against. Quanta Inference lets you point the same harness at text, image, audio and embedding models through one OpenAI-compatible API key, which keeps the case set and the scoring code identical when you compare models or providers (https://quantaintelligence.ai/inference).

Once the basic gate works, three extensions pay for themselves.

Track the flip rate. Run 20 cases three times each and count how many change answers. Cases that flip repeatedly should be split into two clearer cases or fixed in the descriptions. A flip rate you accept becomes the tolerance in your accuracy check, so the number is derived rather than guessed.

Add a second-turn dataset. Single-turn selection is easier than selection after a tool result comes back. For agents that chain calls, record the first call, the tool result, and the expected second call. This is where tool orchestration abuse lives, and red-teaming guides treat multi-turn attacks as a distinct class worth testing separately from single-turn ones (Fiddler).

Keep a case for every retired tool. When you remove a tool, keep one case whose request would have matched it and change the expected label to refuse_no_tool. That case protects against the model retaining a ghost tool from prompt history or examples, the exact failure When2Call was built to measure (When2Call, arXiv).

The broader principle from red-teaming practice applies to selection too: findings should become runtime guardrails and continuous monitoring, not a report that gets filed once (Fiddler). A confusion cell you discover in the harness is a policy question: should that tool be reachable by this agent at all? Answer it in the permission layer, and the harness tells you when the answer is being ignored.