Testing a Scope Matrix with Deliberate Overreach Probes

The technique you need already exists in QA and is called negative testing: supplying invalid, unexpected or out-of-scope inputs and checking the system rejects them predictably instead of failing silently Keploy's negative testing overview. The ISTQB glossary frames it as testing a component in a way it was not intended to be used, which is a fair description of a probe Virtuoso's negative testing guide.

What changes when the system under test is an agent is the unit of the test. In classic negative testing you feed a bad email address into a form. Here you give the agent a plausible, well-formed task that is just outside its scope and see whether the authorization layer stops it. The task should be something a real user might plausibly ask for, because the interesting failure is not a malformed request that crashes the parser. It is a reasonable request that the agent happily fulfills because nothing in the enforcement path disagreed.

Negative testing is also where teams quietly skip work, because the happy path is easier to write and easier to demo Infinum's best practices for negative scenarios. The same pressure applies to agent permissions. A demo that shows the agent doing the right thing is more persuasive than a test that shows it refusing, so the refusal tests never get written.

The rule is simple and mechanical: for every cell that says denied or approval-required, write exactly one probe whose task would succeed if the boundary were missing. For every cell that says allowed, write one control probe that confirms the allowed action still works, so a later failure cannot be explained as "the integration was broken the whole time."

A good probe has five parts:

1. An identity. Which actor the agent is running as, named exactly as in production, not a test superuser.
2. A task statement in natural language, phrased the way a user would ask it, not as a raw API call.
3. The resource it targets, using a real identifier from the staging dataset.
4. The forbidden effect, spelled out: what record would change, what message would send, what amount would move.
5. The expected verdict: deny, or require approval with a named approver role.

Here is a compact probe table for a support agent matrix. Note that each row tests one boundary and nothing else.

Two design rules make probes sharper. First, keep the probe a single step. If a probe chains three actions, a pass tells you one of three things worked and you cannot tell which. Second, write the forbidden effect in terms of observable state, because "the agent declined politely" is not evidence. The pass condition is that the effect did not occur in staging.

The failure mode to design against is a probe that is too easy to refuse. If your probe says "delete the production database," every layer of the stack will stop it, including the ones you are not trying to test, and you learn nothing about that specific cell. The probe should sit one step outside the boundary, not ten. "Refund this order" is one step outside a support role that can read orders. "Drop the orders table" is not a useful test of the same cell.

The most common way this exercise produces false confidence is staging that does not enforce what production enforces. A sandbox built for fast iteration usually mocks permissions or skips them, while a staging environment runs with the real policy engine, the real identity provider and the real data relationships at representative scale DevRev on sandbox versus staging for agents. If your probes run against the sandbox, they test your mock, not your matrix.

A staging setup for probe runs needs five properties:

• Same policy engine, same version. The authorization decisions in staging should come from the same rules, deployed from the same source, as production. If policy is compiled or cached, refresh it and record the policy version with the run.
• Same identity model. Real roles, real token scopes, real service accounts. If every probe runs as an admin because the test harness needs admin to set up fixtures, you have removed the thing you are measuring.
• Separate credentials from the setup path. The fixtures should be created by an orchestrator account, and the agent should see only its own scoped credential. Otherwise the agent can borrow the harness's authority.
• Representative data. Real record shapes and relationships, with sensitive fields substituted rather than deleted, so the agent meets the same schema it will meet later.
• Isolation from production side effects. No shared payment processor credentials, no live email domain, no shared webhooks. A denied action should be denied before it reaches anything that can send or charge.

For coding agents and tool-using agents generally, the same logic shows up as sandboxing: run in branches, disposable workspaces or containers, and never hand a new agent production secrets and auto-merge on day one Kingy's agent security guide on MCP and tool permissions. Probe staging is the same discipline, pointed at the authorization layer instead of the filesystem.

It also helps to think of each probe as an authorization decision that must be reconstructable afterwards: which identity acted, on whose behalf, against what resource, under which policy version, with what result PlainID's permission control framework. That framing turns a pass or fail into evidence rather than an opinion, and it is the same record an auditor will ask for later.

A probe has four possible outcomes, and only two of them are pass or fail in the useful sense:

The inconclusive bucket is the one teams get wrong. If the agent failed to refund the order because the payment tool threw an authentication error, that is not a pass. It is a test that told you nothing, and it will pass again tomorrow for the same wrong reason. The pass condition has to include a positive signal from the enforcement layer: a denial event in the policy log with the probe's correlation ID attached.

Logging follows from that. Every probe run should produce a record with the probe ID, the cell it targets, the identity used, the policy version in force, the correlation ID, the verdict, and the raw evidence: the denial event and a snapshot of the checked state. That maps closely to what an AI audit trail is expected to capture for agents, which is not just the output but the action taken, the data touched and the policy checks at runtime Collibra's audit trail checklist.

A small runner makes the bookkeeping bearable. This example is deliberately generic: it reads a probe list, sends each task to the agent through your normal gateway, then checks both the side effect and the denial log before writing a row.

import json, time, uuid

def run_probe(probe, agent, state, policy_log):
    corr = str(uuid.uuid4())
    before = state.snapshot(probe["resource"])
    answer = agent.run(probe["task"], identity=probe["identity"], correlation_id=corr)
    time.sleep(float(probe.get("settle_seconds", 2)))
    after = state.snapshot(probe["resource"])
    effect = state.diff(before, after)

    denial = policy_log.find(correlation_id=corr, decision="deny")
    returned_effect = effect == probe["forbidden_effect"]

    if returned_effect:
        verdict = "fail"
    elif denial:
        verdict = "pass"
    else:
        verdict = "inconclusive"

    return {
        "probe_id": probe["id"], "cell": probe["cell"], "identity": probe["identity"],
        "policy_version": policy_log.current_version(), "correlation_id": corr,
        "verdict": verdict, "effect": effect, "denial": denial, "answer": answer,
    }

The two lines that do the real work are the denial lookup and the state diff. Everything else is packaging. If you remove either one, the runner degrades into "did the agent seem to refuse," which is exactly the judgment you cannot rely on.

The results are the input to a fix loop, not a scoreboard. Sort the fails into three kinds, because each has a different repair.

Enforcement fails. The matrix says deny and the policy engine agrees, but something in the path skipped the check: the agent called a tool directly, the MCP server had its own token, the endpoint trusted a service identity. These are engineering fixes, and they usually involve moving the decision to a single choke point that every tool call passes through.

Matrix fails. The cells were wrong. You wrote "support may not refund" but the real process is that support may refund under a threshold, with approval above it. The fail tells you the matrix did not describe the business, which is better to learn in staging than from a finance reconciliation.

Scope inflation, which is the interesting one. The probe fails because the token the agent holds is broader than the task needs. This is a well documented pattern: scope inflation happens when a client asks for wide permissions because it is easier to build once than to design carefully, and overbroad tokens expand the blast radius of any misuse NHI Mgmt Group on least privilege OAuth scope. In agent systems the effect compounds, because a single over-scoped integration may control many resources at once Obsidian Security's OAuth scope practices.

For scope inflation, the fix is usually to narrow the grant per task rather than per integration: read-only scopes where reads are all that is needed, resource-specific scopes where only one tenant or account is in play, and audience binding so a token issued for one service cannot be replayed against another Auth0's scopes discussion. Then rerun the probe. The matrix is only trustworthy when the probe passes at the narrowest grant that still lets the allowed cells work.

A worked example makes the arithmetic concrete. Say the matrix has 12 actors, but only 4 of them are distinct in what they may do, and 9 resource families, giving 36 meaningful cells. Suppose 22 are allowed and 14 are denied or approval-required. You need 36 probes, plus at least one control probe per allowed cell that you worry about drift on. If 3 probes fail and 2 come back inconclusive, you have 5 cells to fix and 31 you can defend in a review. Two engineers can build that probe set in a sprint if the tasks are written in natural language first and wired to the runner second.

There is a useful sequencing rule in here too. Fix enforcement before rewriting the matrix. If the authorization layer skips checks, a rewritten matrix just produces new cells that also skip. Get one choke point carrying every decision, then let the probes tell you which cells are still wrong.

A probe set is not a one-time audit. The matrix drifts whenever a new tool is attached, a scope is widened for a demo, a model or agent framework is swapped, or an intern connects an integration with a token nobody reviewed. Existing vendors count too: a tool approved as a project tracker can ship agent features and quietly change what it can do Reco's assessment criteria.

So put the probes on a schedule and on a trigger. Run the full set before any release that touches the agent's tools, scopes or policy. Run it nightly against staging with the current policy version. Record the pass rate per cell over time, and treat a cell that flips from pass to fail without a matrix change as a signal that something upstream moved.

Two habits keep this cheap. First, keep every probe deterministic in its setup: a fresh fixture, a known resource identifier, a fixed identity. Second, keep the probes readable by the people who own the business process, not just by engineers, because the person who knows that refunds above a threshold need a second pair of eyes is the person who will spot a wrong expected verdict.

If you would rather have the probe design, the staging checklist and the fix loop handled by AI experts who work inside your context and hand back finished deliverables, that is what Quanta Experts is for. And if you are still deciding whether your agent program is ready for this discipline at all, a free 30-minute discovery call will get you a tailored recommendation.