A Permission Diff for Tool Schemas: Claim vs. Behaviour

The diff has four fields per row, and you need all four to make a defensible call.

Three verdicts cover almost every case:

• Green: claimed allow and observed allow, or claimed deny and observed deny.
• Red: observed allow where the vendor claims deny. This is the only exit-blocking verdict.
• Amber: observed deny where the vendor claims allow. Not a security failure, but your integration will break if you depend on that row.

Amber is worth logging separately because it also catches stale documentation: a vendor that scopes down without telling you has silently removed a row your agent relied on.

Run only one probe per row per pass. If you batch several calls into one identity, a single allow poisons the whole row and you cannot tell which operation leaked.

Use a dedicated synthetic identity per role, never a shared service account. A role is a bundle of permissions and a scope is a flag on a token, so probing one role through one identity keeps the diff attributable P0 Security. Each identity should hold exactly the scopes the vendor's table grants that role, nothing more Auth0.

A minimal harness run looks like this:

probes = load_rows("vendor_permissions.yaml")
for row in probes:
    token = mint_token(row.identity, scopes=row.claimed_scopes)
    if row.method != "read":
        continue  # never probe writes against production
    resp = call(row.method, row.url, token=token)
    observed = classify(resp)  # "allow" or "deny"
    write_result(row, observed, introspect(token))

Four rules keep the harness safe to run:

1. Probe reads first. Writes, sends, payments, and deletes get staged with a canary account and a human watching, or skipped and marked untested.
2. Call the token introspection endpoint after each probe and store the response, so you have a record of what rights the token actually carried Scalekit.
3. Classify by status code and body shape, not by your agent's interpretation. A 200 with an empty list is a different verdict from a 200 with another tenant's data.
4. Run in a pre-production environment that enforces the same permission boundaries as production. A sandbox that mocks permissions answers the wrong question DevRev.

Synthetic identities fit a formal non-human identity inventory: credential type, owner, purpose, connected systems, permission scope, and expected behavior pattern Sweet Security. Your harness rows are the expected behavior set. When the observed pattern leaves it, you have both a security signal and an audit artifact.

Store results as one record per row per run, then compare runs. The output you want is a short, sortable table a reviewer can act on without opening a notebook.

Two mechanics save time. First, diff against the vendor's machine-readable schema when they publish one, not against prose. A YAML or JSON scope list diffs cleanly and tolerates reformatting. Second, version the claim file yourself, with a commit hash per run, so a red row can be traced to the documentation version it contradicted.

The audit angle matters here. An auditor wants to know which identity acted, on whose behalf, against what resource, under which policy, and with what result PlainID. One record per row, with the token introspection response attached, satisfies that for every documented permission. The EU AI Act's logging requirements for high-risk systems, Article 12, point in the same direction: automatic recording over the system's lifetime so operation can be traced Collibra.

If your agent runtime is Quanta Inference, one API key covers the model calls, but the permission diff lives in your tool layer and runs against the vendor regardless of model. Keep the harness outside the agent.

A red row is the moment the vendor's documentation stops being evidence. Handle it in four steps.

1. Reproduce with a second identity in the same role, same scopes. If both hit allow, it is not a fluke of one credential.
2. Confirm the token carried only the claimed scopes, via introspection. If your token already had a wider scope, the finding is in your own grant, not the vendor's authz.
3. Capture the request, response headers, and a redacted body. Redact tenant data, keep the status code and the field names.
4. Open a ticket with the vendor that asks one question: is this row intended, and will it be corrected or documented?

While the ticket is open, add a compensating control so the release is not blocked by an open question with no end date. The cheap controls, in order of preference:

• Deny at the agent layer. Put a policy check in front of the tool call so the agent cannot reach the row even if the vendor would allow it P0 Security.
• Scope the token down further than the vendor's minimum. If the row is only needed for tenant A, put the tenant ID in the resource indicator so the token is bound to a named resource NHI Mgmt Group.
• Turn the read into a watched operation, with an alert on access to the wider surface.

A vendor that grants more access than documented through delegating privilege also risks audit failure, because permissions assigned at deployment record nothing about the decision to use them PlainID. Your compensating control is what turns an unexplained widening into a recorded decision. Never accept a verbal "we'll fix it" as the control. Either the vendor explains the row in writing and you re-run, or the control stays.

If several vendors are red on the same class of row, that is a procurement signal, not a testing problem. Third party exposure clusters into identity paths, data paths, software supply chain paths, and operational paths, and the identity path is where your diff sits CyberProof. Vendor tiering should raise the row: a vendor that holds payment or customer export rows with a red probe merits a deeper review than one holding a read-only calendar row.

Re-run on triggers, with a periodic floor:

The floor exists because documentation can change silently and because a vendor's authorization logic can change without a schema change. Pick a cadence your team can actually hold, likely monthly for payment and export rows and quarterly for the rest, and record the date of the last clean run per vendor. When someone asks whether a vendor row is safe, the answer is the date and the verdict, not a link to a table.

Ownership matters as much as cadence. Every synthetic identity needs an assigned human owner accountable for its scope, monitoring, and retirement Sweet Security. Without an owner, a red row gets filed under the vendor and never closed. Without retirement, a synthetic identity outlives its harness and becomes standing privilege, the exact failure that agent permission frameworks are built to expose PlainID.

One last habit: keep the last three runs per vendor. A row that was green, then red, then green tells a different story than a row that was always red, and only the history shows you which.

If you are standing up the program and want the identity, logging, and policy layers reviewed against your actual vendors rather than a template, a free 30 minute discovery call with Quanta Consulting can map the harness to your tools: https://quantaintelligence.ai/calendar.