Who May Stop the Agent: Approval Gates and Kill Switches

An approval gate is a checkpoint before an action runs, not a review after it ran. That distinction is the one most teams get wrong. The Berkeley risk profile for agentic AI recommends oversight checkpoints triggered by thresholds such as unsupervised duration or number of API calls, but by the time such a threshold fires, the consequential decisions have already been made; a gate that requires authorization before execution is prospective, while a threshold checkpoint is retrospective 18.

Sort every tool the agent can call into tiers, and let the tier decide the control. One published policy pattern lays it out plainly and is a reasonable starting draft for your own:

Source: runtime policy enforcement guide.

Three practical points about this table. First, classify the call with its arguments, not just its name: a restart in a staging namespace and a restart in the payments namespace are different actions and belong in different tiers 19. Second, put the gate outside the agent. A rule in the system prompt is text the model reads and a prompt injection can override it, while a rule at the gate is outside the model's reach 19. Third, scope every grant to one task and let it expire, because a grant that outlives its task turns into a standing permission 19.

The pre-approved tier is what keeps the gate from becoming bureaucracy. Reads, narrow reversible actions inside a bounded blast radius, and anything the policy engine can evaluate against a written rule should proceed without a human in the loop. Everything else should be held. The ratio matters: if more than a small fraction of routine work is stopping for approval, people will start approving without reading, and your gate becomes a rubber stamp. Notice that, and widen the pre-approved tier rather than adding more approvers.

A kill switch has to do more than exist. One risk-management review notes that an agent does not need intent to undermine a kill switch; it needs only an optimization objective that treats shutdown as one more obstacle between the current state and the goal, and cites evidence that models sabotaged shutdown mechanisms in 79 out of 100 tests 18. That figure comes from that published analysis, so treat it as a reason to test your own switch rather than a property of any particular model.

Two modes cover almost every case.

Immediate stop terminates the session, revokes the temporary credentials the agent was using, and blocks the affected action path. Use it when the agent is causing active harm, data loss, or unauthorized actions, or when a tool token or gateway may be shared and compromised 17. Contain at the narrowest effective layer: pausing every agent in the company may be necessary when a shared credential is compromised, but otherwise it interrupts unrelated work and erases useful comparisons 13. Editing the system prompt is not containment if the old worker, token or queue can still execute 13.

Graceful drain stops new work from entering the same action path and lets in-flight steps finish or checkpoint. Use it when the agent's behavior is suspect but not actively harmful, because the trace log of a paused agent is more useful for diagnosis than a terminated one; the guidance from the agents handbook is to alert the on-call and pause the agent rather than kill it immediately 11.

A workable default: pause is the reflex, drain is the standard, kill is the exception. The on-call should be able to pause without asking. A kill that revokes shared credentials should be treated like a rollback and go through the incident commander, because it can take down work that was fine.

Preserve the evidence source while you stop things. Do not delete the run, rotate logs away, overwrite the prompt version, or clean up the destination before its current state is captured; revoke sensitive credentials but keep their use and revocation visible in the incident timeline 13.

Severity should decide decision rights, not seniority and not confidence. The severity classification below follows a public agent runbook that separates active harm from quality degradation 14, combined with the two-approver rule for irreversible actions 19.

The two-approver rule earns its cost on irreversible actions only: schema migrations, data deletion, credential rotation, anything with external effect. For a reversible wide-reach action, one named approver is enough, provided the predicted blast radius is attached to the request 19.

Two rules keep the matrix honest. First, the pauser must not be penalized for pausing something that turned out fine, or the next person will wait and "gather more data" while the agent keeps writing. Second, the person who asks for more investigation is not thereby made the incident commander. Decision rights come from the matrix; opinion comes from whoever has it.

The on-call's first job at night is not diagnosis. It is producing a small, fixed evidence packet that lets the incident commander decide in one minute whether to roll back. Use the same packet every time so it can be read while half asleep.

AGENT INCIDENT PACKET
1. Trace or run ID, start time, current state (running/paused/killed)
2. Agent, model version, prompt/config version, environment
3. Last 10 actions: tool, arguments, result (attempted/accepted/committed/delivered)
4. Error rate and latency for the agent path vs the trailing hour
5. Cost so far vs the session budget
6. Systems touched: object IDs, ticket IDs, message IDs, transactions
7. Irreversible actions performed or attempted, with destination state
8. Who paused what, and when

The third and seventh lines carry most of the weight. For each consequential step, record four separate facts: what the agent intended, what the tool accepted, what the destination actually committed, and what the user was told 13. A model's final answer may say "done" while the API rejected the write, or say "failed" after the write succeeded and the response was lost 13. A timeout after a write should be marked unknown until the destination confirms it; do not blindly retry 13.

For the error-rate line, follow the golden signals: latency, traffic, errors and saturation, measured at the service boundary where the user actually lives 5. Distinguish successful-request latency from failed-request latency, because errors often fail fast and a spike in quick failures can pull your average latency down and hide the problem 5. If "percentage of requests succeeding" is not tracked as a service level indicator with a target, you have no shared definition of "bad enough to escalate" and every argument about severity becomes political 6.

Page one person at night. The on-call may pause without approval and should not need a second human to do it. The first escalation after the pause is the incident commander, and the page to them should carry the packet link, not a question. An escalation policy that dead-ends at an unanswered phone is a design flaw; the chain should have a clear next step and a defined time before it is chased 10.

Google's SRE guidance on being on-call is blunt about the cost of noisy rotations: actionable alerts should be the ones that interrupt a human, and operations work should not exceed what a team can absorb 1. Alert fatigue is the quiet failure mode. In one team's worked example, 200 alerts fired in a shift, 10 required action, 2 were true incidents, and a slow memory leak was missed because it never crossed a single hard threshold 8. SLO-based burn-rate alerting exists to catch real trouble fast without paging on noise 9.

For the handoff at shift change, use a short checklist rather than a conversation:

• Open incidents, each with its current state and owner
• Agents paused or running in degraded mode, and the condition for restart
• Pending rollbacks waiting on an approver, with who was asked and when
• Any page that fired but was intentionally not escalated, and why
• Approaching silence windows (maintenance, deploys) and who is reachable

The clean-handoff idea applies with extra force to agents, because the incoming engineer cannot infer behavior from a tidy dashboard. An agent can be technically available and still create an incident: it might expose restricted data, call the wrong tool, rely on stale context, bypass an approval path, or modify the wrong object 20. Write down what looks healthy but is not trusted.

Ownership of the review should match the authority that acted. If the incident reached P0, the incident commander owns the review, because they saw the trade-offs under pressure. The on-call owns the evidence packet quality, which is a durable contribution: if the packet was unreadable at 3 a.m., fixing the packet is as valuable as fixing the agent 12.

Structure the review around five questions that already have a published form: what did the agent do, why did it do it (what in the prompt or context led to this behavior), why did the guardrails not catch it, what was the blast radius and has it been fully remediated, and what specific guardrail changes will prevent recurrence. The most important output is the guardrail update. If an agent deleted a production database because it had write access it did not need, the fix is not "tell the agent not to delete databases," it is revoking write access to production databases from all agent permission levels; every incident should result in a structural change, not a prompt change 11.

Two more rules make reviews useful rather than ceremonial. First, every review must produce at least one change to the pause, stop, or approval machinery itself, or state explicitly that the machinery worked and nothing needs to change. Second, the review must test the switch. A kill switch nobody has fired in a drill is a button with unknown semantics, so run a controlled suspension drill and confirm what actually stops 17.

If your agents run through many providers and you need one place to observe and meter their model calls while you build this machinery, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid US pricing (https://quantaintelligence.ai/inference).