Auditing a Manifest Change: What an Agent May Do Differently

Before you read a single value, sort the diff. Every hunk is either widening (the agent gains reach), narrowing (the agent loses reach) or cosmetic (no change in reach). Refuse to review the diff until that sort is done, because reviewers who read top to bottom spend their attention on the loudest line rather than the most consequential one.

The classification is not a formality. Widening changes need a named owner and a bound. Narrowing changes need a regression check. Cosmetic changes need thirty seconds. When a reviewer sees the label next to the hunk, the argument about it being a small change disappears, because the label already answered it.

Two subtleties are worth calling out. First, a widening can hide inside a narrowing patch: a tool gets removed while the remaining tool's arguments get looser, and net reach grows. Second, identity fields are authority fields. If permissions attach to a stable agent identity rather than to a session or prompt, as the manifest guide recommends, then an edit to that identifier quietly re-points every grant. Treat identity edits as widening until proven otherwise.

The most common review failure is approving a tool list and skipping the schemas. Tool design is where silent failure modes get introduced, and narrow, job-specific tools with clear input and output schemas outperform broad wrappers on reliability and governance alike, per these tool use best practices. A schema that accepts a free string accepts anything, including values the reviewer never imagined.

Work through the diff argument by argument and write down what became legal:

• New parameter with a free string type. This is a widening even if the tool name did not change. Ask what the string is used for and whether an enum can replace it.
• Enum extended with a new value. A widening proportional to what that value unlocks. Adding read to a scope enum is not the same as adding write.
• Bound raised (max length, max bytes, max rows, max spend). A widening with an arithmetic answer. Note the old and new numbers in the review.
• required list shortened. A widening, because the agent can now omit a field the downstream system may treat as all.
• additionalProperties enabled. A widening, because unknown arguments now pass validation instead of being rejected.

The mechanical version of this check is easy to automate. JSON Schema with additionalProperties: false rejects unknown parameters, and validation pipelines that check schema, constraints and permissions before execution are the pattern laid out in this tool validation walkthrough. If the manifest schema is not strict today, add that strictness before you widen anything else, because it turns every future review into a shorter list.

# minimum argument check: reject unknown params and assert enums stay closed
def audit_tool_schema(old, new):
    findings = []
    for param, spec in new.get("properties", {}).items():
        if param not in old.get("properties", {}):
            findings.append(("WIDENED", f"new param {param}"))
        elif spec.get("type") == "string" and "enum" in old["properties"][param] and "enum" not in spec:
            findings.append(("WIDENED", f"{param} enum removed"))
    if not new.get("additionalProperties", True) and old.get("additionalProperties", True):
        findings.append(("NARROWED", "unknown params now rejected"))
    return findings

Manifest churn is mostly text. Display names, descriptions, disclaimers and instruction blocks account for a large share of diffs, and on most platforms they genuinely do not change what the agent can call. Microsoft's agent manifest reference splits a manifest into descriptor metadata, agent definitions, and skill groups with inputs and settings, a useful reminder that only part of the document is authority-relevant (Microsoft's agent manifest reference).

But cosmetic is a narrower category than it looks. An instruction edit can widen reach even though the tool list is untouched: telling the agent it may proceed without confirmation, giving it permission to retry on failure, or listing an escalation contact all change behavior. Interface choices like when confirmation is required reveal how the builder intends oversight to work and teach the user where their authority ends, as the Data & Society agenda notes. So apply a simple rule: a text edit is cosmetic if and only if it cannot change a tool call. If the sentence contains may, skip, retry, always or without, it is not cosmetic. Read it as authority.

The payoff for this discipline is attention. If you label formatting and display-name edits as cosmetic in the first pass, the second pass can focus entirely on the handful of lines that change reach, which is what a reviewer's limited attention is for.

The checklist collapses to four questions. Keep the review thread to these and reject attempts to re-litigate the whole manifest in a patch review.

1. What can the agent now do that it could not before? List each widened capability in one sentence, with the tool and the argument that enables it.
2. What can it no longer do? List narrowing changes and any multi-step skill that depended on them.
3. Which arguments and values became legal for each tool? Attach the enum, bound or free-string answer to each.
4. What evidence shows the new authority was exercised as intended? Attach the dry run output and the log lines.

Approve only when every widened line has a named owner and a bounded scope, every new argument is constrained by an allowlist rather than a free string, and the dry run shows the agent hitting the new limit and failing safely. If one of those three is missing, the answer is not yet, not approve and follow up. The whole reason the review exists is that follow-ups get dropped.

A worked example makes the arithmetic concrete. Suppose a support agent's manifest gains a refund tool with an amount parameter typed as a free string, and the approval gate is left optional. The capability is new (widening), the argument is unconstrained (no allowlist), and the owner is not in the diff. That patch fails all three tests. The fix is not a longer review: it is an enum or numeric bound on amount, a required approval above a threshold, and the name of the person who owns the change. That version reviews in minutes.

A dry run is not a demo. It is a pre-production exercise that tests identity, permission scope, policy enforcement and audit logging under production-equivalent conditions, including deliberate tests of denial paths, a distinction this runtime control guide makes explicit. Functional testing confirms the agent produces correct output. A dry run confirms the agent's access is scoped, enforced and logged before it operates for real.

The guide lists four checkpoints worth copying into your review template: agent identity, permission scope enforced at the tool-call layer, policy enforcement with both allow and deny paths tested, and audit logging that records identity, tool and outcome. It also names the recurring gaps those checkpoints catch, several of which are exactly what a manifest diff introduces: permissions granted at the connector level instead of the action level, and audit logs that capture successful actions but omit denied attempts.

For a manifest change, the dry run has one specific job beyond the general checkpoints. It must exercise the newly widened edge. If the diff raised a rate limit, run a burst and watch the new limit trigger. If it added a free-form parameter, submit a value outside the intended set and confirm it is rejected. If it removed an approval gate, confirm the agent proceeds and that a reviewer can still see it did so. A dry run that only replays happy-path traffic proves nothing about the changed authority, which is the only thing this review is about.

Guardian-style supervisory systems run a continuous loop of observe, interpret, decide, intervene and record decision logs, and they escalate when an action crosses risk, financial or ambiguity thresholds (Sweet Security's overview). Your dry run should produce evidence that loop works for the new authority. The logs are that evidence. If the denial was not recorded, the control does not exist for audit purposes.

Authority creep is rarely one big grant. It is a tool here, a scope there, and a bound raised when a demo broke, and none of them individually justified a conversation. Permission model guidance makes the same point from the security side: long-lived credentials baked into agent configuration, inherited user OAuth scopes and effective superuser privileges are the classic anti-patterns, and the fix is scoping permissions to capabilities rather than resources and issuing short-lived credentials tied to a specific execution plan (Auth0 on agent permission models). Apply that to the diff: a widened capability should carry who owns it, what it is bounded to, and when the grant expires or gets re-justified.

The same instinct appears in autonomy design. A delegated agent should receive a bounded objective and an operating envelope covering tools, data, spend and duration, with automatic expiration, and giving an agent a task should not mean giving it all the user's permissions (MIT's five levels of autonomy). A manifest diff that widens the envelope is the moment to reset the expiration. If the original grant had a review date and the widening did not update it, the grant is now permanent by accident.

Use the autonomy levels as a tiebreaker when a diff is hard to classify. Ask what role the human now plays for the affected capability. If the change moves the human from approver to observer for an action that spends money or writes to production, it is a level change disguised as a config edit, and it deserves a conversation rather than a comment.

Runtime enforcement also has a counterpart you should check before merge: whether the enforcement point is separate from the decision logic. When policy decision and enforcement are conflated into one component, it becomes hard to test denial paths in isolation, and that is one of the gaps the dry run guide lists. Reading that split at review time tells you whether a denial will actually stop the call or merely be logged.

Where several teams share agents and models, the review is shorter when every widened capability points at a governed resource rather than a personal key, and Quanta for Teams gives each person an API key inside one shared credit pool with a private skill library and admin controls.

The checklist is only as good as the gate that enforces it. Three practices make the difference between a ritual and a control. First, require the four answers in the pull request description, not in a comment thread, so the evidence travels with the change and shows up in the audit trail. Second, label each hunk in the review interface itself; most version control systems let reviewers annotate lines, and the label is what makes the widening owner obvious a year later. Third, keep a short blocklist of diff shapes that always need a second reviewer: new tools that spend money, identity edits, removal of an approval gate, and any bound raised without a number in the description.

There is also a staffing point. A gate fails when the only person who can read it is on holiday, so at least two reviewers per team should be able to classify a manifest diff unaided. That is a training problem, not a process problem, and it is cheap to solve while the agent fleet is small. The cost grows with every agent you add, and it grows fastest for the agents nobody remembers approving.

Finally, record what you approved. The capability manifest guide recommends a controlled review and versioning process for scope changes so the audit trail reflects when scope changed and why, rather than a silent update buried in code (capability manifest guide). A review thread that answers the four questions and names an owner is that record. The next reviewer, the next incident, and the next audit all read it.