Auditing an Allowlist: Finding Rules No Agent Fires

You cannot audit what you did not record. Before anything else, confirm each policy decision emits the fields below. If your engine already produces them, map the fields. If it produces an OpenTelemetry span per tool call, check that the authorization decision is an attribute rather than free text.

The policy-version field is the one teams skip and later regret. Edit the allowlist three times during a thirty-day window, then join hits against the current file, and every recently added rule shows zero hits and looks dead. Pin the window to one policy version, or tag each hit record with the version live at decision time.

Policy-as-code setups already return a structured decision you can log. The AI SDK's OPA integration consults a .rego policy before every tool dispatch and returns one of four normalized decisions: allow, deny, requires-approval, or not-applicable (AI SDK docs). That fourth value matters for the audit: no rule matched and the SDK default applied. A file whose rules mostly produce not-applicable because of a name mismatch governs nothing.

Keep allows and denies in one log with one schema. Gateway projects make this explicit, logging every allow and deny centrally (Coder). Splitting allows into an application log and denies into a security log guarantees the join misses half the picture.

A minimal record

One structured line is enough. Put the rule id first so a group-by does the counting.

{
  "rule": "allow-git-read",
  "decision": "allow",
  "tool": "git",
  "args": ["log", "--oneline", "-20"],
  "agent": "agent-42",
  "policy_version": "9f3c1a7",
  "ts": "2026-11-02T09:14:31Z"
}

The join is one query: group hits by rule over the window, then left-join that result onto the rule list. Every rule appears at least once, with a count of zero if it never fired. Do not use an inner join. The dead rules are the ones that disappear, and they are half the point.

-- hits_by_rule: rule_id, hits, distinct_tools, distinct_arg_heads
-- rule_list:    rule_id, file_line, intent
select
  r.rule_id,
  r.file_line,
  coalesce(h.hits, 0) as hits,
  coalesce(h.distinct_tools, 0) as distinct_tools,
  coalesce(h.distinct_arg_heads, 0) as distinct_arg_heads
from rule_list r
left join hits_by_rule h
  on h.rule_id = r.rule_id
order by hits asc;

Reading the output is triage, not judgment. The security literature on rule hit counts makes the point cleanly: a low or zero count can mean dead policy, a misplaced rule, a broken data source, or a control that no longer matches conditions, while a high count can mean a useful rule or noisy logic that needs refinement (NHI Mgmt Group). Hit counts are a usage metric, not a quality score. They tell you where to look.

Worked example. An allowlist of 60 rules produced 12,400 decisions in thirty days. The join shows 22 rules with zero hits, 9 above 500 hits each, and 29 in between. The dead rules are 37 percent of the file. The 9 hot rules account for 8,900 decisions, 72 percent of volume. That distribution is typical of a file built by unblocking, and it says where the work is: the top 9 rules are where a missing argument check turns into real risk.

A caveat on shadowed rules. A zero count can mean the rule is unreachable rather than unused. Firewall practice hits exactly this: a stateful rule with a pass action generates no alert log by default, so it never appears in hit counting unless logging is added to it (AWS Security Blog). Before deleting a zero-hit rule, check the ordering. If a generic rule earlier in the file always matches, the specific rule below it is invisible to the metrics. Narrow the generic rule rather than deleting the specific one.

Dead rules are the cheapest finding and the most misleading to act on. A rule that never fires is shadowed, misaddressed, or obsolete.

Shadowed means an earlier, broader rule always wins. Misaddressed means the tool name, path, or command in the rule does not match what the agent calls, which is common after a server is renamed or repointed. This matters because a component's name says nothing about what it does at run time, and an approved connector can be repointed without changing its identifier (Backslash Security). Obsolete means the workflow is gone.

The retirement procedure that breaks nothing:

1. Confirm zero hits against a single pinned policy version.
2. Check ordering for a broader rule that may be shadowing it.
3. Move the rule from allow to deny in audit mode, so the policy records what it would block without blocking.
4. Watch that audit output for one full business cycle. Any hit means the rule was needed and shadowed.
5. With no hits, delete the rule and note it in the changelog.

Step 3 mirrors how mature enforcement tools are tested. BeyondTrust's agent policy workflow runs a new policy in audit mode, replays recorded activity from a past window to see what it would have matched, then publishes to block only after that check (BeyondTrust). Replaying a recorded window turns a deletion into a reversible experiment.

Two guardrails. Do not retire a rule just because its scenario is rare: a break-glass rule that fires twice a quarter is preventive, and hit counts show a control is exercised, not that it is well scoped (NHI Mgmt Group). Tag rare-but-intended rules with an intent comment and exclude them from the dead-rule query. Second, never retire a rule in the same window you changed the tool it governs, or you cannot tell which change caused the zero.

The hot rules are the dangerous ones, because a rule that fires a lot is doing a lot of work under one name. A rule called allow-git-read that fires 4,000 times a month and covers status, log, diff, show, blame, and fetch --all is not one rule. It is a rule plus an invisible decision that reads and writes a remote.

The remedy is argument checks that constrain the rule to the invocations you intended. The OPA example encodes this when it auto-allows only status, log, diff, and show for git while denying push outright as requiring human review (AI SDK docs). The check belongs in the policy file, not the prompt, because the downstream system must enforce authorization rather than the model's judgment; OWASP's Excessive Agency entry calls this complete mediation (NHI Mgmt Group).

To find the missing check, add two derived columns to the join: distinct tools per rule, and distinct leading arguments per rule (the first argument of a command, the HTTP method, the top-level path segment). Any rule with more than a handful of distinct leading arguments has no specified intent. Pull ten sample argument sets for that rule and read them. If you cannot state the one behavior the rule permits, a reviewer cannot tell a good use from a bad one.

Watch the interaction with engines that default to allow. The AI SDK normalizes a missing rule to not-applicable and treats that as allow unless the policy sets a default deny, so a broad allow rule that always matches means every tool you add later is silently covered (AI SDK docs). A default-deny posture makes the join trivially complete: every allowed call carries a rule identity and there is no "everything else" bucket to reason about.

The middle band is where judgment lives. A rule with a moderate count across several distinct behaviors is doing the job of three rules while looking healthy. Splitting it pays compound interest, because each resulting rule becomes independently auditable.

A worked split, using counts from a hypothetical thirty-day window:

The split surfaces a finding the original rule hid: delete and webhook calls are rare and irreversible, and they sat inside a rule that is otherwise read-heavy. Once split, those two rules can carry requires-approval without slowing the 590 calls that dominate volume. The same logic applies to the components you govern. Backslash Security's policy object table separates agents, models, MCP servers, MCP tools, skills, hooks, and rules files, and notes that a trusted MCP server can still expose a destructive tool (Backslash Security). An allowlist with one entry per server rather than per tool is a compound rule by construction and should split at the tool level first.

A rule of thumb: if the behavior in the rule's intent comment needs the word "and," it is two rules. If it needs "or," it is probably a bug.

The cost is real. Splitting 60 rules into 95 raises review effort on every change and can push the decision point toward a file nobody reads. Split only where behaviors differ in risk class or approval requirement. A rule covering six read-only variants of one command is fine as one rule; a rule covering reads and deletes is not.

A single audit is a snapshot. Value comes from a scheduled join tied to policy version. Three practices make it work.

First, make the dead-rule query a gate in continuous integration. When the policy file changes, rerun the join over the last stable window and fail the build if zero-hit rules rise past a threshold. Combined with OPA's opa test, which checks the policy independently of the SDK, you get two layers: the tests prove rules do what they say, the join proves they are used at all (AI SDK docs).

Second, set a review interval on every rule at creation. Firewall governance frameworks require removal of dormant rules after a defined period, and hit counting exists so that requirement can be evidenced (AWS Security Blog). CIS guidance for firewall hygiene is blunt about the mechanism: enable hit counts, then review unused rules and remove or update them (Tenable).

Third, decide what you accept as drift. Some churn is normal: tools get approved, workflows retire. The number to watch is rule count against decision volume. If the file grows while volume stays flat, the join shows the accumulation as dead rules, and you caught a hygiene problem before it became an incident.

For teams running the same audit across many workloads and model providers, the logging and join step is what scales badly by hand. Quanta Inference (https://quantaintelligence.ai/inference) puts text, image, video, audio and embedding models behind one OpenAI-compatible key, which keeps the decision schema consistent no matter which model the agent called.