When an Agent Fails at 2 A.M.: A Runbook for Agent Incidents

Before you change a setting, answer one question: is this one conversation, one tool, one customer, or everything? The severity taxonomy in the incident response chapter is a practical anchor. The top class means the agent took an action that affected production systems, corrupted data, or exposed sensitive information, and needs a human within minutes. The next class means wrong output passed automated checks but a person caught it, and needs investigation within the hour to find out why the checks missed it. The third class means the agent failed obviously and needs a ticket, not an emergency.

Scope also tells you whether you face one agent or the whole platform. Powercode Group's containment guidance is precise here: choose the narrowest action that reliably stops harm. Pausing every agent may be right when a shared credential or tool gateway is compromised, but otherwise it interrupts unrelated work and destroys the comparison data you need. Equally, editing the system prompt is not containment if the old worker, token, or queue can still execute.

Open the incident record and write four things down: the run ID, the time range, the agent and model version you were running, and the prompt or configuration version ID. That last field is the one teams most often forget to log, and it is the field the whole rollback step depends on.

Tool calls are where agents meet reality, and they are the most common source of a 2 A.M. page. An aggregate success rate hides the problem: a 96 percent overall figure can be one tool at 40 percent and eight tools at 100 percent. Break it down by tool name, and read success rate next to latency.

Use the four golden signals from Google's SRE book as your frame: latency, traffic, errors, and saturation, measured at the boundary with the user. For an agent, "the user" means the request to your agent endpoint, and the tools are internal dependencies. A guide to the golden signals adds the rule that matters when reading tool latency: track successful and failed latency separately, because fast failures pull the average down and make a broken dependency look healthy. Percentiles describe the shape, averages hide it.

Three concrete checks:

1. Error rate per tool, last 15 minutes versus the same hour on prior days. A tool that failed 1 percent then and 30 percent now is your suspect. A tool already at 5 percent is background noise.
2. Timeout share versus hard error share. A spike in timeouts usually means the dependency is slow or saturated. A spike in 4xx errors usually means the agent is sending bad arguments, which is an agent problem, not a dependency problem.
3. Retry depth. If your client retries tool calls, count attempts, not logical calls. An agent looping on one tool shows up as a climbing retry count while unique calls stay flat. The incident chapter recommends a kill switch trigger when the same tool call repeats more than three times without progress.

# pseudo-query, adapt to your backend
group_by(tool_name) {
  success_rate   = count(status="ok")      / count(all)
  timeout_share  = count(status="timeout") / count(all)
  client_err     = count(status=~"4xx")    / count(all)
  p99_latency_ms = p99(latency_ms)
  retries        = sum(attempts) - count(logical_calls)
}

If one tool is the outlier, you have a decision rule: a single failing tool is a containment problem, not a prompt problem. Disable that tool for the agent, or point it at the cached or degraded path, and leave the prompt alone. Editing a prompt because a dependency is down is how teams create a second incident. If every tool is failing, look at the credential, the gateway, and the egress path before you look at the model.

Context problems are the quietest failure mode in the whole runbook. A deep dive on context windows puts it directly: once the window is full, earlier information gets silently dropped, the agent keeps running with incomplete information, and it produces confident but wrong results. The failure does not announce itself as an error.

The arithmetic explains why agents hit this so fast. A workflow of 50 steps that carries 20,000 tokens per call accumulates about 1 million tokens of context across the run, because each call carries the original instructions plus everything produced since. A simple chat turn does not. And even when you stay inside the limit, models attend unevenly across long contexts, so material buried in the middle can be effectively lost.

Check four numbers before you change anything:

• Peak context tokens per call, as a percentage of the model's limit. Set a warning threshold well below the ceiling and a page threshold above it. The exact percentages depend on your model and task, so derive them from your own baseline rather than guessing.
• Truncation events. If your context assembler drops or summarizes messages, count those events. Any truncation inside a single user task is a candidate root cause.
• Tool output size per call. One tool returning a large document can consume most of the window in a single step. This is the most common cause of a sudden context jump.
• Tokens per completed task against the trailing baseline. A step change here means a prompt or a tool changed behavior even if nobody touched the code.

The decision rule: truncation with a stable prompt and a stable model means fix context assembly, not the prompt. Compress tool outputs, summarize history earlier, or move large retrieved documents behind a reference the agent fetches on demand. Roll the fix to a canary before the whole fleet, because context changes alter behavior broadly.

Instrument this once and you stop paying for it every incident. OpenTelemetry gives you one vendor-neutral set of APIs and SDKs, with traces, metrics, and logs correlated by shared context that propagates across your request path, so a span for a model call can carry its token count and prompt version next to the tool spans it triggered. Its Collector then routes that data to whatever backend you already run, which matters at 2 A.M. because you read one trace instead of opening four tools.

Guardrails fail in two directions, and the on-call engineer has to tell them apart in minutes. A guardrail that stops firing is a safety problem. A guardrail that fires too often is an availability problem, and it is much more common during an incident because a degraded agent produces strange inputs.

Read the guardrail log with three questions:

• Which rule fired, and on what content? Log the rule ID, the decision, and a redacted excerpt. Without rule IDs you cannot tell a policy change from a traffic change.
• Is the firing rate inside its normal band? Compare the last hour to the same hour on prior days. A rule that normally fires rarely and now fires constantly is either a real attack or a broken detector.
• What did the agent do next? A blocked call that the agent retries in a loop burns budget and produces nothing. A blocked call that falls back to a safe answer is working as designed.

This is where structured logs earn their keep. New Relic's logging guide is blunt about the reason to standardize: key-value logs are searchable, filterable, and analyzable at scale, while free-text logs force fragile regex parsing that breaks when someone rewords a message. SigNoz's examples show the same event in both forms and why the structured one lets you group by field instead of guessing where a value sits in a sentence.

Make the guardrail log use the same field names as the rest of your telemetry. OpenObserve's production field guide calls inconsistent field naming the property teams most often skip and the one that breaks cross-service debugging, and recommends a shared schema enforced across services with fields for service, level, event, timestamp, trace ID, and request ID. Add agent_run_id, prompt_version, tool_name, model, and token_count to that schema the same way. A practical structured logging guide makes the same point with a table of query types: exact match, substring, equality conditions, and group-by are easy with structured logs and somewhere between medium and impossible with plain text.

Decision rule for guardrails: if the firing rate is above baseline and the traffic pattern is unchanged, treat it as a false positive and relax that specific rule for that specific agent, with a time limit. If the traffic pattern changed, treat it as a real signal and tighten. Either way you record which rule you touched and when, because the next shift needs to know.

Most agent incidents that are not dependency failures trace back to a change: a prompt edit, a tool description change, a retrieval config change, or a model routing change. Your runbook needs one question that resolves fast: what is the last known good prompt version, and can I serve it in one action?

That only works if you make three things true before you need them.

First, prompts are versioned artifacts with stable identifiers, stored next to code and deployed through the same pipeline. A prompt that lives in a dashboard and gets edited in place is not rollback-able, because there is nothing to roll back to.

Second, the version is visible in every trace and log line. When you look at a failing run, the prompt version should be a field, not an inference. Powercode Group's timeline guidance lists the configuration version alongside run ID, model version, triggering request, and tool arguments as the minimum record to preserve, and warns against overwriting the prompt version during cleanup.

Third, the reference is mutable at runtime, so a rollback is a config change rather than a redeploy. A deployment rollback takes minutes you may not have if the agent is still writing to a third party.

# pinned agent configuration, illustrative
agent: support-triage
prompt:
  ref: prompts/support-triage@v41   # immutable tag, not a branch
  checksum: 9f2c...                 # assert on load
tools:
  - name: order_lookup
    enabled: true
  - name: refund_issue
    enabled: false                  # re-enable only after the incident closes
limits:
  max_tool_repeats: 3
  max_context_pct: 0.80
  session_cost_budget_usd: 5.00

Roll back one axis at a time, in this order: prompt version first, then tool enablement, then model routing, then retrieval config. Changing two at once means you learn nothing and cannot attribute the fix. After the rollback, confirm three things before you close the page: tool success rate returned to baseline, guardrail firing rate returned to baseline, and no queued job is still executing under the old configuration. That last check is the one that gets missed. A rollback that leaves a worker draining the old queue is not a rollback.

If you run agents across several model providers, keep the rollback path identical regardless of which model served the request. Quanta Inference exposes text, image, video, audio, and embedding models through one OpenAI-compatible API key, so the prompt version, tool policy, and context limits you pin are the same config you flip back to whichever model you were routing to.

Once the bleeding stops, do the verification pass that separates an agent incident from a normal one. The agent response guide gives the frame: separate attempted, accepted, committed, and delivered. An HTTP success may only mean a request entered a queue, and a timeout may hide a committed write. Mark uncertain outcomes as unknown and reconcile with a read-back before retrying anything.

Build a small action ledger: system, object ID, action, known result, evidence link, owner, correction needed. Sort it by reversibility, with payments, messages, permissions, public files, and customer records first. Do not retry a tool call whose outcome is unknown until the destination confirms whether it committed.

Then run the review while the trace is fresh. The five questions from the incident chapter are a good template: what did the agent do, why did it do it, why did the guardrails not catch it, what was the blast radius and is it fully remediated, and what specific guardrail change prevents recurrence. Its most useful rule is that the fix should be structural rather than a prompt tweak: if an agent deleted something it should not have reached, the fix is revoking the permission, not adding a sentence to the prompt.

Close by turning the incident into a permanent check. Every incident should leave behind a new alert, a new eval case, or a new limit in the pinned configuration. If it leaves behind none of those, the next 2 A.M. call will be the same call.

For teams that would rather not build this layer from scratch, Quanta Skills is a library of installable skills you can drop into a coding agent, or put a specialist on the operational work, which is a reasonable way to get tracing and runbook hygiene in place without a six-month platform project.