When the Model Picks No Tool: Scoring the Abstain Case

Before writing any scoring code, write down what each label means in your product's terms. Ambiguous labels produce ambiguous scores.

Two decision rules keep labeling consistent across annotators.

First, if the request can be satisfied only by inventing a parameter value the user did not supply, the label is ask_a_person, not tool_call. When2Call makes exactly this distinction: when the tool exists but required parameters are missing, the expected behavior is a follow-up question, not a fabricated argument (When2Call).

Second, if no registered tool can answer, the label is no_tool even if the request sounds like a tool request. The travel agent that cancels a flight before confirming a rebooking exists is the canonical counterexample: the correct move was to stop and check, then ask (AgentAbstain). Label that case ask_a_person.

For the abstain side, build pairs. Take a case that works today and perturb it minimally: remove one required argument, swap a tool for a similar-sounding one that cannot help, or introduce a contradictory instruction. The clean version stays a tool_call case; the perturbed version becomes no_tool or ask_a_person. AgentAbstain generated its paired set through controlled perturbation of instruction, tool, or environment state, and three annotators rated 94 to 98 percent of sampled tasks as well designed (AgentAbstain). You do not need that rigor to start, but you do need pairs, because unpaired abstain cases let a model game the set by refusing broadly.

Assume you already have a runner that produces, for each case, a trace of tool calls and a final message. The change is small if you keep parsing out of it: classify the trace into an outcome, compare the outcome to the target, and score by class.

TERMINAL = {"tool_call", "no_tool", "ask_a_person"}

def classify(trace, final_text):
    calls = trace.tool_calls
    if len(calls) == 0:
        return "ask_a_person" if asks_question(final_text) else "no_tool"
    if len(calls) == 1:
        return "tool_call"
    return "multi_call"  # keep visible; do not fold into "error"

def score(case, trace, final_text):
    predicted = classify(trace, final_text)
    target = case.target          # one of TERMINAL
    return {
        "case_id": case.id,
        "class": target,
        "predicted": predicted,
        "correct": predicted == target,
        # extra signal for the abstain classes
        "acted_before_asking": target == "ask_a_person" and len(trace.tool_calls) > 0,
    }

Three implementation details matter more than the shape of the function.

Keep a multi_call outcome visible rather than folding it into "error." A model that chains four calls on a one-call task is a different defect from one that picks the wrong function, and collapsing them loses the diagnosis.

Record whether the model acted before it asked. AgentAbstain names post-hoc abstention, where the agent executes an irreversible action and only then acknowledges the problem, as a distinct and more dangerous failure than a wrong answer (AgentAbstain). Your acted_before_asking flag is what catches it.

Do not treat a missing tool call as a failed assertion. Several harnesses, including the widely used lm-evaluation-harness, evaluate tasks by comparing generated output against reference answers or log-likelihoods rather than tool traces (lm-evaluation-harness), which is why teams graft tool logic on ad hoc and then have nowhere to put abstention. Give the abstain classes a first-class return type, not a None.

Report tool-call accuracy as one row, then no_tool and ask_a_person as their own rows. Treat each class as a two-by-two problem: for ask_a_person, the positives are cases that should escalate, and the negatives are cases that should not. Precision tells you how often an escalation was warranted; recall tells you how many warranted escalations you caught. Accuracy alone hides which of the two you are failing, and precision, recall and F1 map directly onto tool correctness and hallucination risk in these pipelines (a developer's guide to LLM evaluation).

Worked example. Suppose your set has 400 cases: 200 tool_call, 80 no_tool, 120 ask_a_person. The model is right on 180 of the tool cases. On the 80 no-tool cases it calls a tool 16 times. On the 120 escalation cases it answers directly 24 times and escalates correctly the other 96. A single pooled accuracy figure is 0.90 plus 0.80 plus 0.80, weighted: (180 + 64 + 96) / 400 = 0.85. Now separate them.

The pooled number is the least useful figure on the page. Whether you ship depends on which misses land on a production system. An unnecessary call to a read-only search tool costs latency; an unnecessary call to a payment or deletion tool does not. That distinction is why access scope should be bounded independently of model behavior: default-deny allowlists and per-persona tool scopes mean a wrong call has a bounded blast radius regardless of what your eval says (Zscaler on least-privilege for agents, Oso on over-permissioned agents).

Also score the paired form. For each pair, count the case correct only if both the clean and perturbed variants are correct, then report that alongside the per-class figures. AgentAbstain found the best agent across 17 frontier models and 4 harnesses reached only 59.5 percent paired accuracy, and that abstention ability was largely independent of general task-solving ability (AgentAbstain). That independence is the finding that should shape your roadmap: improving your tool-call score will not automatically improve the abstain rows.

Per-class scores tell you what happened. A cost matrix tells you what to do about it. Assign a cost to each off-diagonal outcome, in the units your organization actually feels, then compute expected cost per 1,000 cases.

With those numbers, a model that misses 24 of 120 escalations and makes 16 unnecessary no-tool calls costs 24 × 60 + 16 × 1 = 1,456 per run, if every missed escalation is high stakes. A second model that catches every escalation but makes 40 unnecessary read-only calls costs 40. The second model has the lower pooled accuracy in most such comparisons and is the correct choice. This is the decision rule the headline metric cannot give you: weight by where the miss lands, not how many misses there are.

Set the escalation threshold from this table rather than tuning against accuracy. If high-stakes misses cost 60 and unnecessary state-changing calls cost 80, the crossover is close, which is an argument for lowering autonomy rather than tuning the prompt. Autonomy levels L1 and L2, where the agent executes defined tasks but escalates ambiguous or high-stakes decisions, are the target for most production agents (MLflow's tool-use best practices). Your cost matrix is the quantitative justification for that choice.

Build the report so a reader can answer four questions without opening a notebook: which class regressed, in which direction, on which paired set, and at what cost.

Put class accuracy, paired accuracy and expected cost per 1,000 cases on the same page. Add a short "top failure" line per class with one real trace ID. Teams underinvest in abstention because it shows up as a vague error count rather than a named failure with an example attached; naming the post-hoc abstention case next to the trace is what gets it fixed (AgentAbstain).

Two reporting traps recur.

The first is averaging across classes with different sizes. A 400-case set where ask_a_person is 30 percent of examples will have its pooled score dominated by tool cases. Always show per-class denominators.

The second is scoring single attempts. A model that gets the right call most of the time can still fail a multi-step task when errors compound; consistency metrics that measure success across repeated attempts are a better predictor of production behavior than one-shot scores (Splunk's review of agent benchmarks). Run each abstain case three to five times and report the pass rate, not just the last run.

For the abstain classes specifically, add the false-positive side. A model can score well on ask_a_person recall by escalating constantly; the over-refusal side of the act/abstain boundary is a real trade-off, not a rounding error (AgentAbstain). Your no_tool and clean-pair rows are what keep an over-cautious model honest.

If you are building this in-house and want a second opinion on the label taxonomy and cost table before you wire it into CI, a free 30-minute discovery call with Quanta Consulting is a reasonable place to pressure-test the design (https://quantaintelligence.ai/calendar).

Use this as the acceptance criteria for extending the harness. Each item is testable.

• Every eval case carries exactly one target from tool_call, no_tool, ask_a_person.
• At least half the abstain cases are paired with a clean twin that differs by one controlled perturbation.
• The classifier records multi_call and acted_before_asking separately from plain errors.
• The report shows class accuracy, class denominator, paired accuracy and expected cost per 1,000 cases.
• Each abstain case runs at least three times and reports a pass rate.
• The cost table is signed off by whoever owns the downstream system the tools touch.
• Tool scopes are default-deny per persona, so that a wrong call's blast radius is bounded by policy, not by eval score (Oso, Arthur on scoping agent access).

That last item is the bridge between evaluation and engineering. Your eval tells you the model's tendency to abstain; your permission model decides what happens when the tendency is wrong. Both need to be true before an agent with write access goes anywhere near production.