When an Agent Fails at 3 A.M.: A Triage Decision Tree

Run these in sequence and stop at the first failure. Do not reorder them, because each check depends on the one before.

Check 1: does the agent still hold live credentials? List the tokens, keys and short-lived credentials issued to the agent's identity and confirm which are unexpired and unrevoked. An expired token makes the next two checks meaningless, because nothing can fire. A live token with broad scope means the next two checks become urgent. If you cannot enumerate the agent's credentials in under two minutes, that is itself an inconclusive result, and inconclusive means contain.

Check 2: are outbound calls still firing? Look at the gateway, not the agent. Count tool and API calls in the last five minutes against the same window during normal operation. A flat line usually means the agent is stopped or credential-blocked. A steady or rising line means it is acting, and the rate tells you how much time you have.

Check 3: does the state store match the last known good snapshot? Compare what the agent believes about the world against the last snapshot you trust. Divergence in memory, task queue or downstream records is the signature of what one practitioner triage guide calls the sneakiest failure type: an agent that is technically running but producing wrong output, which can damage data long before anyone notices the AgentCenter triage guide. That guide also notes that any case where failure has corrupted state or downstream data calls for a rollback, not an investigation in place.

The decision rule is deliberately asymmetric. Three passes buy you the right to investigate. Anything else buys you a kill switch and a log review, in that order. The cost of an unnecessary containment is a restart and an awkward morning conversation. The cost of a delayed containment is measured in data and spend.

One note on what "acting" means when the press is running. CISA's first-48 guidance for organizations experiencing a cyber incident is built around expectations and sequencing rather than heroics, and it stresses that responders should follow the plan they already approved rather than invent one mid-crisis CISA's response and recovery resources. An agent that can still call a live tool is the agent equivalent of a compromised host. Containment order beats diagnosis quality.

Once you decide to investigate rather than contain, the question becomes whether the failure is local or spreading. The signals that matter are the ones you can count in under a minute.

Four signals tell you it is cascading:

• Blast radius. Count how many other tasks or agents are waiting on this one's output. A single agent failing in isolation is low urgency; the same agent with three others blocked behind it is high urgency regardless of the task the AgentCenter triage guide.
• Retry amplification. A rate-limited agent that retries in a loop turns a quota problem into a load problem. Watch the retry counter, not the error text.
• Shared-state writes. If the agent writes to a store that other services read, wrong writes are a cascade in progress even when the agent itself looks healthy.
• Alert fan-out. One agent should not produce alerts in five unrelated services. That pattern means something downstream is now reacting to the agent rather than the reverse.

If two or more of these are true, you are in a cascade and the containment decision is already made for you. Time spent confirming a cascade is time the cascade keeps running.

Symptom-first alerting helps here. Google's guidance on SLOs argues that alerts should fire on what users experience, with causes monitored for troubleshooting rather than paging anyone, because cause-based alerting produces noise that slows response the Google Cloud SLO post. For agents, the user-facing symptom is usually "something happened that a human did not approve." Page on that, and use the cascade signals to decide whether you are containing one thing or several.

Open logs in this order because each one answers a question the previous one cannot.

1. The gateway access log. This is your ground truth for what actually left the perimeter. It records requests that were made, regardless of whether the agent thought it made them. If the agent process was replaced or restarted, the gateway still has the record.
2. The agent action log. This is what the agent believes it did: tool calls, reasoning steps, decisions. Use it to understand intent, not to establish facts.
3. The credential and identity audit log. Token issuance, scope changes and revocations, with timestamps. This is where you find out whether the agent obtained a credential it was not given.
4. The state store changelog. What was written, when, and by which actor. Use the timestamps to reconcile against logs one and two.

The reconciliation rule is the single most useful thing in this article: put the action log and the gateway log side by side on the same clock, and where they disagree, the gateway wins. An agent that reports it made four calls while the gateway recorded forty is not lying; it is missing context, running a stale build, or being driven by something other than its own code path. Any disagreement is a finding worth escalating, not a logging bug to shrug at.

Google's SRE guidance on monitoring draws a useful line between white-box monitoring, which uses internal metrics and logs, and black-box monitoring, which tests externally visible behavior as a user would see it the SRE monitoring chapter. The gateway log is your black box for an agent. The action log is white box. You need both, and you need to know which one you are trusting when they conflict.

Teams that cannot answer "what did each agent do yesterday and what did it touch" are the common case rather than the exception. As one governance review puts it, most teams know agents are running but few can prove what each one did and whether it stayed inside its drawn lines the Drata agent behavior guide. A gateway log you can query is what turns that question from a shrug into a two-minute lookup.

Write the threshold before the incident, because a threshold invented at 3 a.m. will always be set too high. The rule that works in practice: contain if any of the three checks fails or is inconclusive, or if you cannot complete all three within your first ten minutes on the call.

The ten-minute framing is the important part. It converts "I am not sure yet" from an excuse to wait into a trigger to act. You are not deciding whether the agent is dangerous; you are deciding whether you have enough information to rule that out, and the absence of proof is the answer.

Escalation policies exist to catch silence: unacknowledged alerts move on automatically after a set timeout rather than waiting for someone to notice the Motadata on-call guide. That same guide cites Uptime Institute research finding close to 40 percent of organizations had suffered a major outage caused by human error in a recent three-year window, and that 85 percent of those traced back to a procedure that was ignored or badly written the Motadata on-call guide. The kill switch is exactly the kind of procedure that gets ignored when it has no named owner.

Two containment levers exist, and you should know which one is one command away:

• Revoke first. Kill credentials and the agent becomes a process that can think but not act. This is the least destructive lever and the fastest to reverse.
• Stop second. Terminate the process or disable the scheduler. Use this when the agent holds cached credentials, writes directly to a store, or you cannot confirm the revocation took effect.

# Containment checklist, run in this order and timestamp each step
date -u +"%Y-%m-%dT%H:%M:%SZ"           # log the decision time
# 1. Revoke the agent identity's active tokens (use your IdP's CLI)
# 2. Confirm revocation propagated: check the gateway for calls after the revoke timestamp
# 3. Disable the agent's scheduler entry or scale its deployment to zero
# 4. Snapshot the state store BEFORE any restore, so the timeline survives
# 5. Only then restore from the last known good snapshot

Step four is the one people skip in a hurry, and it destroys the evidence you will want at 9 a.m.

A runbook that has never been walked under pressure is a document, not a capability. Tabletop exercises exist to test the response plan in a low-stress environment by talking through roles and decisions rather than touching systems, and they surface gaps in planning and unclear ownership before a real incident does the Bitsight tabletop explainer. CISA publishes customizable exercise packages with objectives, scenarios and discussion questions, including cybersecurity scenarios you can adapt to an agent failure simply by writing the scenario yourself the CISA tabletop exercise packages.

Two formats fit this problem well. A rapid-fire scenario runs roughly ten to thirty minutes with minimal preparation and lets several people each play responder on the same scenario, which is ideal for drilling the three checks and the ten-minute rule the Sophos tabletop guidelines. A full stakeholder scenario runs two to four hours and pulls in legal, communications and business owners, which is what you want the first time you discover that nobody told the comms team about containment authority the Sophos tabletop guidelines.

Run the rapid-fire version quarterly with a single twist: the facilitator withholds one of the three checks. "The gateway log is unavailable, what now?" The right answer is contain. Teams that have practiced the inconclusive branch reach it in seconds. Teams that have only practiced the happy path argue about it.

Fatigue is part of the human side too. Guidance on on-call design is consistent that a rotation which floods one responder produces slower, less accurate decisions, and that sustainable load sits at roughly two to three actionable incidents per shift rather than a wall of pages the incident.io on-call guide. If your agent fleet pages the same person five times a night, your triage tree is fine and your rotation is not.

A decision tree works at 3 a.m. only if every branch terminates in an action someone is authorized to take. Three design rules make that true.

Name the owner of every node. "Check state store" is not a task unless it says who runs it and how long it may take. Pair each check with a timeout, and let the timeout escalate rather than the responder's judgment the Motadata on-call guide.

Keep alerting narrow. Page on the symptom an agent cannot produce safely: an unapproved action reaching a live tool. CISA's incident response checklist exists so responders work from a short, ordered list rather than a menu of options, which is the same reason your agent tree should have fewer branches than you think the CISA incident response checklist.

Keep the evidence trail separate from the fix. The post-incident phase is where the response plan feeds back into preparation and detection, and it only works if the timeline survived containment the NIST lifecycle overview. Snapshot first, restore second, every time.

If you are standing up agent fleets across several model providers and want the gateway and credential surface to be one thing you can revoke in a single step, that is the problem Quanta Inference is built for: text, image, video, audio and embedding models through one API key, OpenAI-compatible, prepaid and priced in US dollars (https://quantaintelligence.ai/inference). One key per environment means one revocation per environment.