Metering Agent Retries at the Tool Boundary

The boundary is the function in your runtime that turns a tool-call request from the model into an actual side effect and returns a result. Everything crossing that boundary is countable in one place: model-initiated calls, retries from a transport, retries from an agent loop, retries from a framework, and re-dispatches after a restart.

A useful mental model is: the wrapper is a gate. Nothing gets to the tool without passing through it, and every pass through the gate is an observable event. This is also the boundary that idempotency practice points to. As one analysis puts it, retries belong in the orchestration layer and at the tool boundary, not in model output, because the model may propose a retry but code must decide whether that attempt is a new operation (Formation on agent tool retries).

That gives you a clean decomposition:

The wrapper is the only row that has full coverage of the events you actually care about.

The design goal is one counter series per task. A task is a durable unit of work, usually a single user request, with a stable identifier the wrapper can attach to every dispatch. The counter increments on entry to the wrapper, not on exit, so a tool that hangs and times out still produces a count.

A minimal Python version:

from dataclasses import dataclass, field

@dataclass
class TaskMeter:
    task_id: str
    attempts: int = 0
    retries_by_cause: dict = field(default_factory=dict)

    def record(self, cause: str) -> None:
        self.attempts += 1
        self.retries_by_cause[cause] = self.retries_by_cause.get(cause, 0) + 1

def wrapped_tool(tool_name, tool_fn, meter, cause="initial"):
    meter.record(cause)
    return tool_fn()

Then call it from the loop that actually resolves the tool:

def dispatch(task, tool_name, args, tool_fn, meter):
    try:
        return wrapped_tool(tool_name, lambda: tool_fn(**args), meter)
    except TimeoutError:
        return wrapped_tool(tool_name, lambda: tool_fn(**args), meter, cause="transport_timeout")

Three rules that keep this honest:

1. Count at entry, not on success. A retry that also fails is still a retry. Counting only successful invocations loses information about the ones that burned budget.
2. One counter per task, not per tool. Per-tool counters are useful, but they fragment attention when the real question is "how many attempts did this task cost." Keep the per-task counter as the primary key and the per-tool counter as a dimension.
3. Never reconstruct the counter from logs. Logs narrate; they do not count reliably. Sampling, log levels, and truncation all lose events.

The wrapper also becomes the natural place to enforce a retry budget. A turn-level cap that the wrapper refuses to exceed is the same pattern called out in production retry guidance: a turn that retries a dozen times on a misconfigured tool can burn far more than a healthy request, so a budget is not optional (Motomtech on retry budgets).

A counter that only says "five attempts" is not actionable. A counter that says "five attempts: one transport timeout, three validation errors, one model re-plan" tells you what to fix. Classification happens where the wrapper sees the failure, because the wrapper is where the exception, status code, or error payload actually lands.

A practical taxonomy:

The Azure retry storm guidance is emphatic about the failure classification problem: retrying a client-side validation error that will keep failing worsens the problem rather than solving it, and clients should limit both the number of retries and the duration (Azure retry storm antipattern). The wrapper is where this distinction becomes a number you can look at.

The most under-counted class is "model judgment." A model can look at a valid result, decide it is inadequate, and ask for another call. That request is a new operation, not a retry of a failed one, but as a matter of behavior it is an extra tool call that costs money and moves the state of the world. Counting it separately, and never lumping it in with transport retries, keeps both numbers meaningful.

The obvious risk with a wrapper is that you count the same attempt twice. This usually happens in one of three ways.

First, nested wrappers. If a framework already wraps tools in its own retry logic and you wrap again, one dispatch becomes two increments. Decide once which layer owns the counter and make it the outermost boundary.

Second, retries that cross the wrapper boundary. If code above the wrapper catches a failure and re-dispatches the plan, the wrapper sees a brand new dispatch and cannot tell it apart from a genuine new call. That is fine from a counting standpoint as long as the task ID is stable, but the classification will read "initial" unless the caller passes a cause. Make the cause an explicit parameter, as in the code above.

Third, replay after restart. A durable execution log might replay a tool call that already committed, and the wrapper would count it again as a fresh attempt unless it recognizes the task ID and operation key. The idempotency literature is blunt about this: a fresh tool-call identifier is not a stable business-operation identifier, so attach the key outside model-controlled arguments (Formation on tool retries and idempotency). The same identity that makes the effect safe makes the count correct.

A useful policy summary:

If the task_id and operation_key match a completed dispatch:
    record "recovered" and return the cached result
Else:
    record the dispatch with a cause and execute

That single rule de-duplicates both the side effect and the counter increment.

Suppose a task asks an agent to file an expense report. The agent calls create_expense with a receipt. The tool's HTTP client has a 10-second timeout. On attempt one, the request reaches the server and the server commits the record, but the response never makes it back before the timeout.

Here is how the numbers look under three different metering strategies:

The model client sees two requests. The logs, depending on sampling, see between zero and five lines. The wrapper sees five attempts on one task, with a cause distribution of one transport timeout, one transport timeout from a framework retry, two model re-plans, and one rate limit. Only the wrapper view supports a decision like "our expense tool times out on 40 percent of first attempts; fix the server side, not the retry policy."

This example also shows why per-task counting matters more than per-tool counting. Per-tool, you would know that create_expense was called five times across a batch. Per-task, you know that one expense report was nearly filed three times and involves two model re-plans. Without the wrapper, you would have no way to tell them apart.

Log-based metering is the default approach for teams already shipping to an observability stack: emit a log line at each retry, then run a query that counts matching lines. It has real strengths. It requires no new abstraction. It captures arbitrary context for free. It works with vendored SDKs you cannot wrap.

It also fails in specific, predictable ways.

The last two rows are where the choice actually lives. If your team already has a mature searchable log pipeline and treats retries as a secondary signal, log-based counting has a lower operating cost. If retries drive money, rate limits, or support load, the wrapper counter is the primary instrument and logs are the annotation.

OpenTelemetry is a useful middle path. The GenAI semantic conventions define a execute_tool span, and the agent span conventions distinguish tool execution from agent invocation, which means a correctly instrumented tool boundary already produces a per-attempt telemetry signal that a collector can sum (semantic conventions for GenAI agent spans). The caveat is maturity: the agent and tool conventions remain in Development status, so the attribute names and span shapes can change (Greptime on OTel GenAI maturity). Build the counter in code first; map it to semantic conventions second.

For teams running many tools through the model context protocol, the same choice appears at the protocol level. MCP servers expose tools, resources and prompts as distinct features, and the client is the party that sees both the request and the result (MCP specification). If you build a client-side wrapper around every tool the host exposes, you get one counter for all of them, whether the tool is local or remote, and whether the remote server has its own retry logic or not.

Multi-agent systems introduce a coordination problem that a per-task counter exposes but does not solve. One practitioner running nine agents against a single API quota described a chain of 60 or more failed requests in a single incident when every agent retried on the same schedule, producing a second wave of throttling and a third, burning through a shared hourly limit in roughly 90 seconds (Dresher on rate limiting multi-agent systems). The wrapper counter is what makes that wave visible; it is not what prevents it. Traffic-light throttling, priority tiers, and jittered backoff are the prevention.

On the authorization side, per-task counters align naturally with per-user or per-project identity. The MCP authorization tutorial notes that authorization is the mechanism for per-user usage tracking and rate limiting, and that this is only meaningful when the identity is preserved through the tool call (MCP authorization guide). A wrapper that carries the task ID and the calling identity together is set up for both.

Finally, the idempotency contract that makes retries safe is the same one that makes counts trustworthy. Analysts writing on the topic keep returning to the same construction: derive the key from the orchestrator's durable step, not from model output, and keep it stable across transport retries, re-plans, and restarts (tianpan.co on the orchestration contract). If your counter is keyed the same way, the number and the side effect stay in agreement.

When your tasks call many kinds of tools, and the tools call many kinds of models, having one consistent place to instrument helps. The natural next step for teams already wiring metering into a multi-model stack is a single inference path across text, image, video, audio and embedding models through one OpenAI-compatible key, with prepaid pricing in US dollars, which is what Quanta Inference provides.