AI coding

Irreversible Actions an Agent Must Never Take Without a Human

Antonio Evans8 min read

A split scene: on the left, an agent's tool-call panel shows a proposed 'delete production table' action highlighted in red; in the center.
Key points

Key points

  1. 1An agent should never delete data, move money, send messages outside your system or change production state without a human approval that is enforced in code before the tool call runs.
  2. 2The four irreversible categories are destructive deletes, payments, external sends and production changes, and each has verbs and API surfaces you can enumerate.
  3. 3Put the policy check in the tool wrapper, not in the prompt, because a prompt rule is text the model reads and a prompt injection can override it.
  4. 4Deny by default, fail closed when a classifier errors, return a structured refusal and require a signed, single-use approval token before the four categories execute.

Why a prompt rule is not a control

Teams often write a paragraph in the system prompt: "Never delete production data without asking." That paragraph is advice, not enforcement. It works only when the model reads it, understands it, and chooses to comply, and it disappears the moment a tool result carries conflicting instructions.

That failure mode has a documented shape. In a widely cited incident, during a code freeze on a production database, the freeze existed only as a text instruction; the agent ignored it and deleted the data of nearly 1,200 companies (harness engineering write-up). The ban was real. The enforcement was not.

Prompt injection makes this worse because the attack does not need to bypass your infrastructure controls. It steers an autonomous system that already holds legitimate access toward committing the attack itself, using valid credentials the whole way (Teleport). OpenAI's own guidance reaches the same conclusion from a different direction: rather than trying to perfectly classify malicious inputs, design the system so the impact of manipulation is constrained even if an attack succeeds (OpenAI).

That is the design rule for everything below. The model proposes. Your code decides.

The four-category taxonomy

Classify by what happens if the call runs and turns out to be wrong. A read that leaks data is bad, but a delete that runs is gone. The categories below cover the actions where "undo" does not exist or costs more than the approval would have.

CategoryTypical verbsAPI surfaces to watchDefault control
Destructive deletesdelete, drop, truncate, purge, force-push, rm, unlink, overwriteDatabase clients, storage SDKs, git push --force, DROP TABLE, recursive file operationsHard block, two named approvers, restore plan attached
Paymentscharge, refund, transfer, payout, subscribe, invoice, adjust creditPayment processor APIs, billing endpoints, refund and credit toolsHard block, one named approver, amount and counterparty shown
External sendssend, publish, post, email, invite, webhook to third partyEmail and SMS providers, social and CMS publishing APIs, outbound webhooksHold, amount and recipient list shown, single-use token
Production changesdeploy, apply, scale, rotate, migrate, set IAM, disable loggingCloud APIs, Kubernetes admission, CI/CD triggers, firewall and IAM endpointsHold, blast radius attached, two approvers for irreversible steps

OWASP's agent guidance sorts actions into four risk levels running from plain reads up to irreversible or security-sensitive changes, and asks for explicit approval on high-impact or irreversible actions plus an action preview before execution (runtime policy enforcement guide). The table above is that idea made concrete for the actions most likely to hurt.

Two refinements matter. First, classify on arguments, not just tool name: a restart in a staging namespace and a restart in the payments namespace are different actions and belong in different tiers. Second, read, generate and execute are separate capabilities and should receive separate permissions, so a drafting agent that writes an email does not automatically inherit the right to send it (ThreatLocker).

The line for "send" is the boundary of your own system. Writing a message to your internal queue is a one-hop problem you can roll back. Delivering to a customer, a partner or a public feed leaves your blast radius permanently.

Put the check in the tool wrapper

The enforcement point is the code that sits between the model's tool call and the real API. Every tool call goes through one function, and that function holds the last word.

def execute_tool(name, args, context):
    decision = policy.decide(name, args, context)  # may raise
    if decision.effect == "deny":
        return refusal(name, decision.reason)
    if decision.effect == "approve":
        token = approvals.request(name, args, decision)
        return pending_approval(name, token)  # agent sees a structured "wait"
    return real_tool_map[name](**args)        # allow, logged

Three properties make this wrapper a real control rather than a speed bump. It runs inside the process that holds the credentials, so the model cannot route around it by calling another tool. It sees the arguments before execution. And it returns a value the agent can read, so the agent learns to wait instead of retrying. An approval gate is an authorization primitive, not a UX affordance (Scalekit).

The same reasoning applies to the shell. Terminal agents need a command grammar rather than a regular expression list, and the check has to see the command before it executes (harness engineering). The OWASP guidance is explicit that a rule at the gate sits outside the model's reach while a rule in the system prompt does not (runtime policy enforcement guide).

Framework hooks help but do not replace this. The OpenAI Agents SDK runs an input tool guardrail before the tool executes and can skip the call or raise a tripwire, but those hooks run inside the agent's own process (runtime policy enforcement guide). Pair an agent-side hook with a policy gate outside the agent and with infrastructure backstops such as admission control and cloud IAM, which still hold when an agent-side check is skipped.

Deny by default and fail closed

Deny by default means an unknown tool, an unparseable argument, an unrecognized destination or a missing policy entry all resolve to refusal. This is the opposite of how most agent stacks are wired, where a new tool is usable the moment it is registered.

The permission model that works for agents is stated flatly: the LLM proposes actions and policy authorizes them, because letting the agent's reasoning become your authorization logic is how you build an incident generator (Oso). Over-permissioning usually arrives quietly, through connectors added to make a demo work, and it never gets tightened.

Fail closed covers the unhappy path. If the classifier times out, if the approval service is unreachable, if the policy engine returns an error, the answer is refuse and log, never allow and hope. A useful test of any gate: unplug the policy service and confirm that writes stop rather than proceed.

def decide(name, args, ctx):
    try:
        rule = policies.get(name)
        if rule is None:
            return Deny("no policy for tool")        # deny by default
        return evaluate(rule, args, ctx)
    except Exception as e:
        return Deny(f"policy error, failing closed: {e}")  # fail closed

Two more defaults earn their place. First, scope grants to one task and set an expiry, because a grant that outlives its task turns into a standing permission (runtime policy enforcement guide). Second, give each agent its own identity with a named owner, so a bad call is attributable to both the agent and the person who authorized it.

Return a structured refusal, not an error string

When the gate blocks, the agent needs to understand what happened or it will loop, retry, or hunt for another tool that does the same thing. Return a typed object with a stable shape.

{
  "status": "refused",
  "category": "destructive_delete",
  "tool": "db.delete_rows",
  "reason": "irreversible: requires two approvers and a restore plan",
  "approval_url": "https://…/approvals/9f2c",
  "expires_in_seconds": 900,
  "retryable": false
}

Three fields do the heavy lifting. retryable: false tells the agent to stop rather than iterate. approval_url points at the human queue entry, so the agent can surface "waiting on approval" to its user instead of inventing a different route. reason carries the policy rationale, which is the context a reviewer will also need.

Approval as a workflow state, not a sentence in the prompt, is the pattern the stronger implementations converge on: the run pauses, a reviewer sees the proposed action and its effect, and the run resumes only after a valid decision (Promptessor). Approve and reject are usually too narrow; modify and escalate are useful decision types, and reviewer identity and authorization have to be checked, not assumed.

Approval tokens: signed, single-use, bound to the call

The gap most approval implementations skip is what happens between the pause and the resume. The credential that was valid at pause time is not guaranteed to be valid at resume time, and a queue-then-execute model has to hold credential state across that window while the agent references an identity and connection name, never a raw token (Scalekit).

A token that carries the decision should therefore be:

  • Signed, so the tool wrapper can verify it was minted by your approval service and not by the agent.
  • Single use, consumed on first execution, so a replay of the same token cannot run the delete twice.
  • Bound to the exact call, including a hash of the tool name and arguments, so an approved refund of one amount cannot authorize a refund of ten.
  • Short lived, expiring in minutes, so an approval granted on Friday does not authorize a merge on Monday.
  • Bound to an approver identity, so the audit trail records who said yes and what they were authorized to approve.
def consume(token, name, args):
    claims = verify_signature(token)            # raises on bad signature
    assert claims["tool"] == name
    assert claims["args_hash"] == sha256(canonical(args))
    assert claims["exp"] > now()
    assert store.mark_used_once(claims["jti"])  # atomic, fails on replay
    return True

Then revalidate state before you execute. A pending action can go stale while it waits: the record changed, the balance moved, the deploy target moved. If the world no longer matches what the reviewer saw, request a new decision rather than honoring an approval for a situation that no longer exists (Promptessor).

Approval fatigue is the quiet killer of this whole scheme. Constant prompts train reviewers to click yes without reading. Start with materially consequential actions, watch the telemetry, and only loosen thresholds for repeated, well-understood actions inside tighter boundaries (ThreatLocker).

What the reviewer must see

An approval prompt that says "Allow this agent?" protects nothing. The reviewer needs enough context to judge the specific call: the agent, the requesting user, the tool, the target resource, the data involved, the requested permission and the expected result, plus the proposed diff, recipients or privileges where relevant (ThreatLocker).

A worked example makes the economics clear. Suppose a remediation agent proposes a schema migration touching a table with 40 million rows, and a restore takes about four hours versus a human review that takes about ten minutes. Add the two-approver rule and you have spent twenty minutes of reviewer time to bound a four-hour outage plus whatever data the migration dropped. That ratio, not a blanket rule, is how you decide where approvals go.

For irreversible and security-sensitive tiers, the stronger pattern is to refuse rather than hold inside the agent flow and require the change to be made outside the agent entirely. IAM changes, disabling audit logs and promoting from staging to production belong there (runtime policy enforcement guide). Some actions should never be one click away from an autonomous loop.

Rollout checklist

Work through this in order; each step is testable on its own.

StepDo thisPass condition
1Inventory every tool that can write, and the credential each holdsNo tool writes with a broad shared credential
2Tag each tool with a category from the fourEvery write has a category, every read is logged
3Route all calls through one wrapperRemoving the wrapper breaks tests
4Turn on deny by defaultUnregistered tool returns refusal
5Add fail-closed behaviorKilling the policy service blocks writes
6Emit structured refusalsAgent stops, surfaces approval state, does not retry
7Mint signed single-use tokensReplay and tampered-argument tests fail
8Build the human queue with reviewer contextReviewer sees action, effect and reversibility
9Add expiry and revalidationStale approvals are rejected
10Log proposal, decision and executionAudit can answer who approved what

The audit line is not optional for regulated teams. SOC 2 requires evidence that logical access controls were functioning at the time of each action, and most security reviews want the stronger claim that a human authorized this specific operation at this timestamp. Scope grants do not satisfy that; approval events do (Scalekit).

If you are wiring several model providers and tool surfaces behind one gate, a single key across text, image, video and embedding models removes one moving part from the credential inventory. That is what Quanta Inference is for. For the reviewer side, the context an approver needs is the kind of thing a mapped expert can assemble and hand back as a finished decision packet, which is what the experts at Quanta Experts are built to produce.

FAQ

FAQ

Can I rely on the model to ask before it deletes?

No. A rule in the system prompt is text the model reads and a prompt injection can override, while a rule at the gate sits outside the model's reach. The documented failure is an agent that ignored a code freeze stated only as instruction and deleted the data of nearly 1,200 companies. Treat prompt instructions as explanation, never as the control.

What counts as an external send?

Anything that leaves your system boundary: email to a customer, SMS, a post to a public feed, an outbound webhook to a partner, an invite to an external address. Writing to an internal queue is a different category because you can recall it. The test is whether the recipient is outside your control and whether delivery is reversible.

How do I stop reviewers from rubber-stamping everything?

Route only consequential actions to humans and keep the rest automatic, then use telemetry to tune the thresholds. Vague prompts like "Allow this agent?" train mechanical approval, so show the action, the effect, the reversibility and the evidence. Repeated, well-understood actions can eventually run inside tighter bounds rather than keep asking.

Why not just use OAuth scopes and be done?

OAuth answers whether the agent may call this category of operation; it cannot answer whether this specific invocation should execute, with these arguments, right now. A single scope covers both a sandbox field edit and a bulk update of production records. Approval events also give you the human-at-a-timestamp evidence that audits ask for and scope grants do not.

What should happen if an approval arrives after the situation changed?

Reject it and request a new decision. Revalidate the target's current state against what the reviewer saw before you execute, and if they differ, treat the approval as stale. Pair this with short expiries and single-use tokens bound to the exact arguments, so neither replay nor drift can slip a changed call through an old yes.

Sources

Sources

  1. Harness Engineering, AI Agent Permissions as Policy in ...
  2. How to Prevent Prompt Injection in AI Agents
  3. Designing AI agents to resist prompt injection
  4. How to implement runtime policy enforcement for AI agents
  5. What should require human approval when using AI agents?
  6. Human-in-the-Loop Tool Calling: Approval Gates for AI ...
  7. How to Prevent Over-Permissioned Agents
  8. Human-in-the-Loop AI Agents: Approval Workflows & Best ...
About the author

About the author

Antonio Evans
Antonio Evans

Antonio Evans is the founder of Quanta Intelligence and a fractional CTO who builds HIPAA-ready AI systems for health and life-science companies. Before that he was technical co-founder of FriendsEAT for twenty years, CEO of Socialdraft, and a data engineer at IBM. Antonio lives with his dog Santino in Lisbon, Portugal.

All articles