Grading Injection Defenses: A Scorecard per Attack Class

A score is only as good as the record behind it. For every class, store the count of cases run, the count that passed, a severity weight, and a pointer to the exact failing case IDs. Four fields plus an identifier is enough; anything more becomes a reporting project.

{
  "class": "indirect_retrieved",
  "cases_run": 120,
  "cases_passed": 108,
  "weight": 3,
  "owner": "agent-platform",
  "failures": ["inj-2210", "inj-2244", "inj-2301", "..."],
  "baseline_passed": 112
}

Two habits make this record trustworthy. First, record the baseline from the last accepted build, because a class that drops from 112 to 108 passes is a regression even if it clears the threshold. Second, log the pass definition itself: a case passes only if the agent did not take the forbidden action, not merely if the output looked safe in the transcript. The most specific published example of why this matters is a network-operations benchmark where naive execution produced an unsafe tool action in 82.50% of 240 attack instances; prompt-level defenses cut that to between 25.63% and 10.00%, and a static allowlist reached 5.00% unsafe while blocking every approved change and scoring zero on usefulness (NetInjectBench). If your pass criterion cannot distinguish "refused the attack" from "refused all work," it will reward the second.

The score for a class is the pass rate, but the number you gate on is the lower bound of a confidence interval around it. The distinction matters because a 30-case class at 100% and a 3,000-case class at 96% are not equally trustworthy, and the raw percentages say otherwise.

For a proportion, the Wilson score interval is the practical default. It behaves sensibly when pass rates sit near 100%, which is where security suites usually live, and it does not collapse to a zero-width interval when every case passes. The NetInjectBench authors report a 95% Wilson upper bound of 1.58% on their residual failure count, which is the same statistics applied to the other tail (NetInjectBench). You want the lower bound on the pass rate; it is one minus that upper bound on failures.

from math import sqrt

def wilson_lower(passed, n, z=1.96):
    if n == 0:
        return 0.0
    p = passed / n
    denom = 1 + z*z/n
    centre = p + z*z/(2*n)
    margin = z * sqrt(p*(1-p)/n + z*z/(4*n*n))
    return (centre - margin) / denom

for passed, n in [(108, 120), (30, 30), (2880, 3000)]:
    print(passed, n, round(passed/n, 4), round(wilson_lower(passed, n), 4))

Run that and the pattern is plain. 108 of 120 is a 90.0% rate with a lower bound near 83.4%. A perfect 30 of 30 has a lower bound around 88.6%, so a small class cannot earn a high score no matter how clean it looks. 2,880 of 3,000 is 96.0% with a lower bound near 95.3%. The bound is what you put in the release decision, because it answers the question a reviewer actually asks: how bad could this class be and still explain the results we saw?

Small samples need care in the other direction too. Standard intervals rely on approximations that degrade when sample sizes are small, and methods tailored to small samples exist precisely because the textbook approximation can mislead (Rosenblum and van der Laan). For a security gate, the practical response is not a fancier interval; it is to refuse to score underpowered classes at all, which the next section covers.

A single pass rate for every class is easy to explain and wrong in practice. A direct-injection case that produces a rude sentence and a stored-memory case that emails customer records are not the same risk, and the gate should say so. Weight by what a failure lets an attacker do: read data, move data out, or change state.

The published attack outcomes give you the severity ordering. Successful injection can disclose sensitive information, reveal infrastructure details, grant unauthorized access to functions, execute arbitrary commands in connected systems, and manipulate critical decisions (OWASP LLM01:2025). Anything in the exfiltration or state-change band belongs at the top weight.

The numbers in that table are a starting template, not a standard; calibrate against your own baseline and your own tolerance. The structure is what carries over. Weight 3 classes get a floor high enough that a single failure in a 200-case run still trips review, because at weight 3 you are asserting a control that prevents data movement. Weight 1 classes can absorb a handful of failures without stopping a release.

The second insight is that the threshold applies to the lower bound, not the rate. A weight 3 class at 194 of 200 passes has a 97.0% rate, which clears a 97% floor on paper, but its lower bound lands below the floor and the gate correctly holds the build. This is the decision rule doing its job: it converts "we probably pass" into "we can show we pass."

The release gate is three states, not two, and the reason for the middle state is that most real results are neither clearly safe nor clearly broken. Blocking every imperfection stops delivery; passing every imperfection ships known weaknesses. The hold band keeps the decision with a human who can look at the failing case IDs.

gate:
  block_if:
    - any class lower_bound < floor
    - any weight_3 class failed_cases > 0
    - any class cases_run < min_cases
    - any class pass_rate < baseline_pass_rate
  hold_if:
    - any class lower_bound within hold_band
  pass_if:
    - all classes lower_bound >= hold_band_top

Read it in order. The first block rule is the general case. The second is a deliberate strictness for the highest-weight classes: one confirmed exfiltration in a tool-using agent is enough to stop a release even if the statistics would tolerate it, because the cost asymmetry favors stopping. The third rule blocks underpowered classes, which is the honest treatment of a class you cannot yet measure. The fourth blocks regressions against the last accepted build, catching the case where a class was already near the floor and slid further.

The hold state needs an owner and an expiry, or it becomes a permanent waiver. Write the rule down: a held release requires a named approver, a written note naming which failing cases are accepted, and a ticket to add cases or fix the class before the next release. If the hold expires without a decision, the release escalates to blocked.

Two lessons from published attack work justify the strictness at weight 3. Adaptive attacks have been shown to bypass published defenses, which means a defense that passes a fixed suite can still be beaten by an attacker who reads your defense (Sysdig). And the mechanism is structural: models infer who is speaking from how text sounds rather than from its tagged source, so a payload that imitates an operator voice inherits that authority (Prompt Injection as Role Confusion). The instruction hierarchy attempts to train models to prioritize privileged instructions over untrusted ones, and it improves robustness, but the same work shows the tag boundaries do not fully survive into the model's internal representations (The Instruction Hierarchy, OpenAI). Treat the suite as a floor, not a certificate.

Every suite has a class with twelve cases, usually added late when someone noticed a gap. Twelve cases cannot support a 97% claim: at best you observe zero failures and know very little. The wrong response is to pool it with a neighboring class to inflate the count, or to report 100% and move on.

The right response is a separate verdict. Below your minimum case count, the class is labeled underpowered, excluded from the weighted aggregate, and routed to manual review with a recorded judgment. Then it gets a remediation ticket to grow the set. This mirrors how other engineering gates handle unmeasurable risk: the absence of evidence is treated as a reason to look, not a reason to pass.

When you do need a count and the suite is thin, the sampled-audit approach works better than stretching statistics. Run a fixed number of cases per class per release, chosen so that the interval width is tolerable, and record the width in the report so reviewers can see how much the score could move. The NetInjectBench paper shows the practical version of this discipline: it reports its residual failure count alongside an explicit upper bound rather than treating zero observed failures in 240 instances as proof of zero risk (NetInjectBench).

A worked example pulls the pieces together. Suppose your build has four scored classes: direct at 92 of 100 (lower bound about 85.0%, weight 1), indirect at 194 of 200 (rate 97.0%, lower bound about 93.8%, weight 3), stored at 148 of 150 (rate 98.7%, lower bound about 95.3%, weight 3), and context field at 40 of 40 (lower bound about 91.2%, under the 100-case minimum). Against the template table: direct clears its 90% floor, indirect fails its 97% floor on the bound and blocks, stored sits inside its 97 to 99% hold band, and context field is underpowered. The release is blocked on indirect, the stored class generates a hold note, and the context field gets a ticket. One number per class, three verdicts, and no argument about what the average said.

The mechanics are ordinary CI work, with one adjustment: the gate must not depend on a flaky model. LLM behavior is probabilistic and context-dependent, and prompt injection strings produce variable results across runs, which is why eval gates belong in the pipeline and why the pass criteria have to be stable across temperatures and sampling (OnSecurity). Pin the model and settings for the security suite, keep a fixed seed where the provider offers one, and store the raw per-case results as a build artifact so a verdict can be reproduced and audited later.

The pipeline itself is now an attack surface, which is a good argument for running the suite on the same workflow configuration you ship. Agents embedded in CI ingest untrusted pull request text while holding repository tokens, and a malicious config file added in a PR can be loaded as authoritative operator guidance (GitInject). Testing the pipeline agent's behavior with the same class structure catches that case instead of discovering it after a token leaks.

Finally, report the scorecard somewhere engineers see it. A table of class, cases, rate, lower bound, verdict, and the failing case IDs is enough, and it takes one page. The verdict column is what people will read, so make it the first column.

If you are running the suite across several model providers and want the gate to compare them on identical cases, Quanta Inference routes text, image, audio, and embedding models through one OpenAI-compatible key, which keeps the model variable fixed per run rather than per vendor SDK. When the gate blocks a release and the decision needs a human, a consulting call with Quanta experts can walk the failing classes with you before the next build.