Detecting an Agent That Retries Its Way Around a Limit

You cannot detect what you did not record. Retrofitting is painful, so instrument at the boundary where sessions, tool calls and identities meet. The OpenTelemetry GenAI conventions give you a shared vocabulary for this, and while the agent and tool orchestration attributes are still marked Development, the core chat attributes are stable enough to build dashboards on (OpenTelemetry GenAI registry, OpenObserve guide).

At minimum, emit one event per LLM call and one per tool execution, with these fields:

Two of these fields do the heavy lifting: attempt number and outcome. Without them you can measure traffic but not evasion. With them, a session that opens, calls six tools, fails, and opens again is trivially distinguishable from a session that opens, calls two tools, and resolves.

The agent span conventions formalize the structure: an invoke_agent span wraps a unit of agent work, and each tool invocation gets its own execute_tool span, which now requires the tool name in the span name (OpenTelemetry agent spans). If you emit to that shape, the ratios below become simple span queries.

Raw counts go up when traffic goes up, so they tell you little. Ratios compare the cost of an attempt to the value it produced.

Sessions per resolved task. Count distinct session ids divided by resolved tasks, per identity, per hour. An agent that retries cleanly on the same session keeps this near 1. An agent that opens a fresh session after every failure drives it upward without moving the resolved count. This is the single most diagnostic ratio you can build, and it is the basis for the alert threshold in the next section.

Distinct tools per session. Count distinct tool names divided by sessions. A healthy session uses a small, stable set. A session that cycles through a dozen tools before resolving (or before giving up) is either confused or probing for one that is not throttled. The tool definitions themselves are a schema, and a mismatch between what the model sends and what the server expects produces a validation error rather than a call (MCP schema reference), so a rising diversity count paired with rising schema errors is a strong signal of blind tool cycling.

First attempts to retries. Count first attempts divided by total attempts per session. Below roughly 0.5, more than half of all calls are retries. Some of that is normal during an outage. Retries that stay high for hours, across many sessions for the same identity, while resolved tasks stay flat, is the shape of an agent working around a limit.

A worked example makes the arithmetic concrete. Suppose identity svc-reporting normally handles 50 resolved tasks per hour, with 55 sessions and a success rate of 0.9. That gives 1.1 sessions per resolved task. During an incident it resolves 12 tasks, opens 60 sessions, and its success rate falls to 0.2. Sessions per resolved task is now 5.0, about 4.5 times baseline, and the work it did complete cost five sessions apiece. No single request looked unusual. The ratio is what changed.

Spreading work across identities is the hardest evasion to catch, because each identity's activity is individually below its limit. The defense is to make identity part of the grouping key, not just a filter.

The first thing to check is whether identities are even distinct. MCP deployments commonly offer two authentication shapes: shared authentication, where every user of the agent uses the same identity and user context does not persist, and individual authentication via OAuth identity passthrough, where each user's permissions carry through (Microsoft Foundry MCP authentication). If you run shared auth with one key for everyone, you have no per-user signal to analyze, and an agent can exhaust the shared key while every user sees only their own small usage.

Authorization also gates whether you can track usage per user at all. Per-user usage tracking and rate limiting are listed as reasons to enable authorization on an MCP server, and the flow is OAuth 2.1 with protected resource metadata discovery (MCP authorization tutorial). If your server accepts unauthenticated requests, you have thrown away the identity dimension that evasion detection depends on.

When you do have identities, check for these patterns together:

• Many identities, same source segments (same tool set, same prompt shape, same time window) with low per-identity volume.
• New identities appearing at the same moment the primary identity's success rate drops.
• A drop in total resolved tasks across identities that does not match a drop in total token spend.

That last one is the tell. Evasion spends money without shipping work.

Do not pick thresholds from intuition or from a single incident. Measure a quiet week, then derive numbers you can defend.

1. Pick a baseline window with no incidents and no quota hits. One full week is enough to capture the weekday and weekend shape.
2. For each identity, compute sessions per resolved task, distinct tools per session, and first attempts divided by total attempts, bucketed hourly.
3. Take the median and the 95th percentile for each ratio, not the mean. One bad afternoon skews a mean badly.
4. Set the warning threshold at the 95th percentile. Set the page threshold at twice the 95th percentile, or at the baseline median times three, whichever is higher.
5. Require the breach to hold for two consecutive buckets. This kills single-bucket noise, which is common when a batch job lands at the top of the hour.
6. Re-baseline after any deliberate change in agent behavior, such as adding a tool or changing a retry policy.

As a concrete illustration: if the baseline median for sessions per resolved task is 1.1 and the 95th percentile is 1.8, warn at 1.8 and page at 3.3. The example identity above hit 5.0, comfortably past the page threshold. If your baseline is noisier, say a 95th percentile of 3.0, the same absolute breach means less, and you should prefer the higher of the two page rules.

Pair every ratio alert with an absolute floor so tiny denominators do not page anyone. A session-per-resolved-task ratio of 6.0 on three resolved tasks is noise. Require at least a few dozen resolved tasks or sessions per bucket before evaluating the ratio, and record that floor next to the threshold so the next person knows why it exists.

A retry storm and deliberate evasion look similar in aggregate volume and very different in structure. Use the table below as a triage checklist before you escalate to anyone.

The structural fixes for retry storms are well cataloged: cap attempts and duration, back off between attempts, use a circuit breaker, and honor retry-after (Azure Architecture Center). A traffic-light throttle that reads remaining quota from response headers and parks background agents at low quota is one practical implementation; reading x-ratelimit-remaining and x-ratelimit-reset after every call lets you slow down before you hit the wall (Tamir Dresher). Those fixes reduce retries. They do not stop an agent from opening a new session to reset its own budget, which is why you also need max sessions per identity per hour and max tool calls per session enforced server-side, not by the prompt.

There is a harder version of this problem: an agent whose retries are not its own idea. A malicious MCP server can return tool responses carrying hidden instructions that the model treats as trusted context, causing it to call restricted tools or send data out (the OWASP MCP Tool Poisoning entry). If your ratio alerts fire alongside unexpected calls to privileged tools, stop treating it as a quota question. Apply the recommended controls: allowlist servers, validate response shape, isolate privileged tools, and enforce restrictions at the execution layer rather than in the system prompt.

Work the alert in order, so you know what you are dealing with before you change anything.

1. Pull the identity's last hour of sessions, sorted by start time. Look for the retry-to-new-session pattern: fail, close, reopen.
2. Compute the three ratios for that identity and compare each to its baseline percentile. One breached ratio is a tooling problem. Two or three together is evasion.
3. Check whether new identities appeared in the same window. If so, the evasion is spread and you need the identity grouping, not just the session grouping.
4. Inspect the tool call sequence in the worst session. Repeated calls to the same tool with different arguments means a loop. A climb through unrelated tools means probe behavior.
5. Check for out-of-scope tool calls. Any call to a privileged tool from an identity that normally does not touch it moves this from a quota incident to a security incident.
6. Apply the server-side cap first, then fix the agent's prompt and retry logic. Capping breaks the loop; fixing the retry policy stops it recurring.

The order matters because capping alone tends to push a persistent agent toward a more creative workaround. Capping plus a fixed retry policy, with backoff and a session-reuse rule, removes the incentive.

You can build all of this yourself: instrument spans, ship them to a store, write the ratio queries, and wire the alerts. Many teams do, and the resulting dashboards are worth the effort.

What you should not do is meter per request and hope. The decision rule is simple: meter on outcomes and identity. If a session cannot be tied to a resolved task and a single owner, you cannot see evasion, only volume.

For teams that would rather not run a logging stack for every model they call, routing model traffic through one metered endpoint gives you the per-identity and per-token view that the ratios need, and it works across text, image, video, audio and embedding models with an OpenAI-compatible interface (Quanta Inference). Pair that with server-side caps and the retry discipline above, and the new-session trick loses its payoff.