Sample Rates by Risk Tier: Proving the Audit Was Met

The core decision is which tier each action belongs to. Sort by two questions: can this be undone, and what is the blast radius if it goes wrong?

Two rules keep the tiers honest. First, tier by the action's worst realistic outcome, not its average one: a read that touches regulated personal data is medium at least. Second, never let the agent choose its own tier. The tier is a property of the tool, set in the tool definition and checked by the runtime, the same way approval requirements are attached to a tool rather than inferred by the model Microsoft's agent framework docs. Cloudflare's human-in-the-loop patterns make the same point: an approval gate belongs to a connector or workflow layer, not to the model's judgment Cloudflare Agents docs.

A useful sanity check for a proposed rate: at your current volume, does the tier produce a queue a reviewer can actually finish? If a low-risk tool fires 40,000 calls a week, one in a hundred gives 400 samples, roughly 11 hours of review at 90 seconds each by Beri's timing assumption. If that is too much, cut the tier's rate or narrow what counts as low risk; do not quietly skip samples.

A review that happened in a Slack thread leaves no evidence. Each sampled action needs a record with the same fields every time, written by the system rather than the reviewer. The fields below are the minimum to reconstruct who approved what, under which rule, and why.

Model these on the approval pattern Temporal documents, which carries approver, decision, comments and timestamp as structured data in a signal rather than a bare boolean Temporal approval pattern. Nylas's CLI audit log shows the same shape in a different context: eight structured fields including timestamp, source, status and correlation ID, with sensitive values redacted before writing Nylas audit guide. Redaction matters for your evidence link too: store the reviewer's note and the action summary, not the full payload if it carries personal data.

Write the sample record from the runtime, not from the reviewer's tooling. A reviewer ticking a box in a separate spreadsheet is a second source of truth that will drift.

Proving the audit was met is arithmetic plus a tolerance you set in advance. For each action and period:

1. Count executed actions from the runtime's own logs, per action and tier.
2. Multiply by the sample rate to get the expected sample count. Round half up; state the rounding rule.
3. Count sample records that carry all seven fields and a non-empty decision.
4. Compare. Coverage is logged samples divided by expected samples.

A worked example makes the tolerance question concrete. Suppose a low-risk search tool ran 47,300 times in a week at one in a hundred. Expected samples: 473. Logged with a decision: 451. Coverage: 95.3%. Is that met? If your threshold is 98%, no, and the shortfall is 22 samples you cannot account for. Do not estimate them from the period before. Find out whether the sampling function failed, whether decisions were logged without a reviewer, or whether some calls bypassed the runtime entirely.

Use a stored property rather than an ad hoc guess to pick the sampling mechanism, so the expected count is genuinely deterministic. For the low tier you can sample by hash:

import hashlib

def should_sample(action_id: str, rate: int) -> bool:
    """rate=100 means one in a hundred. Deterministic, independent of order."""
    bucket = int(hashlib.sha256(action_id.encode()).hexdigest(), 16) % rate
    return bucket == 0

assert should_sample("act_9f2c", 100) in (True, False)

The same input always lands in the same bucket, so reruns and retries do not inflate the count, and you can recompute the expected sample set from the action log alone. That last property is the one auditors actually want: the ability to re-perform the selection PCAOB AS 2315.

Tolerance should scale with tier. A reasonable starting point is zero tolerance on high (every action has a record, no exceptions) and a small tolerance on medium and low, documented before the period starts. An unreconciled shortfall is a finding even when every sample you did review passed.

When a sampled action shows a real problem (an unauthorized deletion, a refund outside policy, a customer email with wrong data), the rate for that action, and for its tier if the cause is systemic, moves up one step: low to medium, medium to high, high stays at every action and gains a second reviewer or a hard block. The escalation is automatic because manual escalation is exactly the step that gets skipped during a busy week.

Hold the stricter rate until a defined clean run passes. Define it before you need it, in numbers: for example, 200 consecutive actions or 30 days, whichever is longer, with zero failed samples and no open findings. Then return to the original rate, and record the return in the same log with a reason. The reversibility framing behind this is the same one that decides what happens when an approval times out: internal, reversible actions can default to proceeding, but actions that leave the system or move money should hold and escalate, never auto-approve on silence Orbiresearch on approval timeouts.

A raised rate is a cost you accept, not a punishment. A tool at one in a hundred that fails twice climbs to one in ten and then 100%, which is how Beri's piece recommends treating gates: put expensive human attention on the irreversible few and audit the rest by sample with planted errors to keep reviewers awake Beri. Planting a known-bad action into the medium queue at a low frequency tests the reviewers as well as the agents.

A rate table without reviewer discipline reproduces the rubber stamp at a smaller scale. Three practices help.

Keep the queue short and mixed. Review depth drops as queue length grows and attention decays after long sessions; most teams do better with short, frequent review blocks than long sittings CodeAnt on review fatigue. If the 90th percentile wait for a medium-tier decision exceeds your timeout, the timeout fires on normal days and people learn to ignore it Orbiresearch.

Make rejections cheap and countable. Track rejection rate per tier and per reviewer. A reviewer at zero rejections over hundreds of samples is either lucky or not reading; a reviewer whose median decision time is a few seconds probably is not reading either, which Orbiresearch flags as a different and more serious problem than slowness.

Sample the sample. Once a period, pull the evidence link on a handful of approved records and check that the reviewer's note actually addresses the action. This is the same logic as a code review checklist: a repeatable list of what to check reduces cognitive load and catches omissions that open-ended scanning misses CodeAnt.

If your agents call tools across many models and providers, one place to land the runtime that records the fields above is a single inference endpoint; Quanta Inference (https://quantaintelligence.ai/inference) routes text, image, video, audio and embedding models through one OpenAI-compatible key, which keeps the action log in one schema.

Before the policy goes live, each item should have an owner and a written answer.

• Every tool in production has a tier, with a one-line reason.
• The tier is enforced by the runtime, not suggested in the prompt.
• Each tier maps to a rate, and the rate is stored with each sample record.
• The seven log fields are mandatory; a record missing one is not counted as coverage.
• The selection function is deterministic and re-performable from the action log.
• Coverage thresholds and rounding rules are documented before the period opens.
• The escalation ladder (low to medium to high) and the clean-run definition are written down.
• Timeouts per tier are chosen, with a default action that never auto-approves money movement or external sends.

The first month will surface the mismatches: actions filed under the wrong tier, tools whose volume makes their rate unaffordable, reviewers whose queues exceed their time. Fix the table, not the log. If you need a specialist to build the runtime, the tier enforcement and the reconciliation report with your team, Quanta's experts (https://quantaintelligence.ai/agents) work as a line of business and hand back finished deliverables rather than a design document.