False Positives in AML Alerts: Measuring and Reducing Them

Before you change a single threshold, build a baseline. You need to know what normal looks like so that any later change can be compared to it, and so that a genuine drop in noise is not confused with a drop in risk coverage.

Pull a fixed window. Ninety days is a good default: long enough to include month-end and quarter-end spikes, short enough that rules have not changed underneath you. If your rules changed inside the window, note it, and consider excluding those weeks or picking a cleaner period.

For each rule in production, extract:

• Alerts generated in the window, by rule and by calendar week.
• Alerts closed in the window, by disposition type (closed no action, escalated to case, SAR filed).
• Backlog at the start and end of the window, by rule and by age bucket.
• Analyst minutes recorded against each alert, from your case management system if it captures time, or from a weekly self-reported log if it does not.
• Alerts per analyst per week, so you can see whether the queue is being worked at all.

The backlog is the number people skip. If a rule generates 2,000 alerts a quarter and 400 remain untouched after 60 days, its effective precision is unknown and its risk is hidden. An unworked alert is not a controlled alert, whatever the policy says. Regulators expect a risk-based approach with full audit trails on every disposition as Sumsub's transaction monitoring guide sets out, and a swelling backlog is one of the first things an examiner will probe.

A simple throughput table keeps the baseline honest:

Two things fall out immediately. Rule A is the biggest consumer of analyst time, and Rule C is the cheapest per alert. Until you know which of them actually produces confirmed suspicion, you cannot say whether either is worth its cost.

Sampling is where measurement programs usually die. Either nobody does it, or somebody reviews 12 alerts by memory and declares a rule "noisy." A documented sample with a written rubric survives audit, staff turnover and a change of opinion.

Design the sample

Use simple random sampling within each rule. If a rule generated 1,800 alerts in the window, a sample of 60 gives you a precision estimate with a margin of error of roughly 12 to 13 percentage points at 95 percent confidence when precision is near 10 percent. That is coarse. For a rule you intend to defend to an examiner, sample 100 to 150. For rules where you are only deciding whether to look closer, 40 to 60 is enough to rank them.

Stratify by disposition if you like, but do not let the sample be drawn only from closed alerts with favorable notes. That biases you toward the alerts analysts already found easy to clear.

Write the rubric first

The rubric is a one-page document with definitions and tie-breakers. Keep it to four labels:

1. True: the alert describes activity that a reasonable investigator would escalate or investigate further, whether or not a SAR was ultimately filed.
2. False: the alert is explained by documented, legitimate customer behavior with no unresolved suspicions.
3. Undetermined: the data needed to decide is missing, stale or inaccessible. This is a data quality signal, not a rule signal.
4. Known: activity already covered by an existing case or a prior SAR on the same customer. This is duplication, not detection.

The distinction between "true" and "SAR filed" matters. If you label by SAR outcome you will conflate rule quality with investigator judgment, and you will never tune anything. If you label by whether the alert was worth a human's attention, you measure the rule.

Two reviewers should independently label the first 30 items before the program scales. If they disagree on more than about a fifth of them, your definitions are not tight enough. Fix the rubric, not the reviewers.

Compute precision and cost per true case

With labels in hand, the arithmetic is straightforward:

precision        = true / (true + false + known)      # exclude undetermined from the base
undetermined_rate = undetermined / all_reviewed
minutes_per_true  = total_rule_minutes / true_count

Worked example using the numbers above. Suppose Rule A's sample of 100 labels resolves to 7 true, 85 false, 5 known and 3 undetermined. Precision is 7 divided by 97 resolved items, or about 7.2 percent. Rule A spent 12,960 minutes to generate roughly 126 true alerts (7.2 percent of 1,800), so it burns about 103 analyst minutes per true case.

Rule C's sample of 60 resolves to 9 true, 50 false and 1 known. Precision is 15 percent. Rule C spent 1,180 minutes to generate about 45 true alerts, so it costs roughly 26 analyst minutes per true case.

Now the comparison is not about volume. Rule A generates more true alerts in absolute terms because it is huge, but it costs four times as much per true case. That is a defensible finding, and it is one you can put in front of a regulator or a budget committee without hand-waving.

The most common tuning mistake is to attack whichever rule generates the biggest number. Volume alone is a poor target, because a high-volume rule with decent precision is doing exactly what it should. Rank by the product of precision gaps and volume, or more simply by the analyst minutes spent per true case. High volume plus low precision is the target. Low volume plus low precision may just be a rule worth retiring. High volume plus high precision is the rule you protect.

A decision table helps make the call explicit:

That last category deserves attention. A low-volume, high-precision rule may be under-alerting, and tuning programs rarely look for that. If a rule fires 30 times a quarter and half of those are true, you may be sitting on a detection opportunity rather than a noise problem.

Two more signals belong in the ranking. First, the undetermined rate. If a third of a rule's alerts cannot be resolved because the customer profile or counterparty data is stale, the fix is a data enrichment project, not a threshold change. Incomplete or outdated customer data is a documented driver of false positives as Lucinity notes in its list of causes. Second, the known rate. High duplication means your case management is not linking alerts to existing cases, and you are paying analysts to rediscover what they already documented.

Each tuning change should be a hypothesis with a predicted outcome. Write it down before you deploy. For example: "Adding the customer's stated business purpose as a suppression condition on the cross-border velocity rule will reduce alerts by roughly 40 percent while cutting true alerts by no more than 10 percent."

The mechanics of change fall into a few reliable categories:

• Threshold adjustment: raise or lower the amount, count or velocity trigger. Cheap, fast, and the first thing everyone tries. It also has the bluntest effect on recall.
• Segmentation: apply different thresholds by customer risk rating or business type, so a payroll processor and a retiree are not judged by the same rule. Risk-based segmentation is the core expectation of modern monitoring regimes as Fraudio's practitioner guide describes.
• Contextual fields: add counterparty history, account age, expected activity or geography so the rule evaluates behavior rather than a single number. Lack of contextual awareness is repeatedly named as a primary cause of false positives per Alessa's review.
• Suppression logic: exclude known recurring patterns such as scheduled supplier payments or payroll runs, with the exclusion itself documented and reviewable.
• Consolidation: merge overlapping rules so one customer event produces one alert and one case, rather than five.

Whatever you change, the rule stays frozen for the measurement window. If you keep adjusting it weekly, you will never accumulate a clean comparison. After 60 to 90 days, re-sample the same way, with the same rubric, and compare four numbers against the baseline: alert volume, precision, true alerts found and minutes per true case.

The comparison is where most programs stop short. A drop in alert volume is not a result. A drop in alert volume with stable or improved true-alert yield is a result. A drop in alert volume with a fall in true-alert yield is a trade-off, and someone senior has to sign off on it, in writing, with the numbers attached. That sign-off is your defense.

Machine learning models can score alerts so investigators work the highest-risk items first, which improves throughput without touching the detection rules at all. DataRobot's accelerator illustrates the pattern: train a model on historical alert data to predict which alerts lead to a SAR, then rank the queue by score as described in their documentation. This is additive and low-risk, because the rule set stays intact and the model only reorders.

Graph and network analytics go further, modeling relationships among accounts, devices and counterparties to find structures that per-transaction rules miss. The Bank for International Settlements' Project Aurora work on graph-based detection is often cited as evidence that new methods can find more laundering while cutting noise substantially as summarized by Facctum. Treat those numbers as directional, not as a promise for your data.

The case for modern detection is real, and the research literature on AI in financial crime argues that static threshold logic cannot adapt to changing behavior as this review in the Journal of Economic Criminology sets out. But every model you add is a model you must govern. Regulators hold institutions responsible for the conceptual soundness of any algorithm their systems use, including third-party ones, and expect documentation detailed enough for a knowledgeable outsider to reconstruct the model as Kaufman Rossin's BSA/AML model risk guidance explains. The OCC's model risk guidance, issued with the Federal Reserve and FDIC, stresses risk-based validation and monitoring and explicitly addresses vendor products per the OCC's announcement.

That has a practical consequence for tuning. If you add an alert-scoring model, the rubric from step two becomes your validation instrument. You can show that ranked-order review surfaced true cases faster, with a fixed sample and a fixed definition. Without that instrument, you have a vendor's slide, not a control.

The measurement program is a control, so it needs the same discipline as the rules it governs.

Document the baseline once, then version it. Store the sampling frame, the random seed or method, the rubric, the reviewer IDs and the raw labels. If a rule change is challenged eighteen months later, this is the file that answers the question.

Keep a change log with predictions and outcomes. One line per change: rule, change type, date, predicted effect, measured effect, approver. This is the difference between tuning and tinkering.

Separate the roles. The person who tunes the rule should not be the only person who labels its alerts. Independence does not have to mean a full second-line review for every change, but at minimum the sample should be reviewed by someone outside the tuning team.

Report the small set of numbers that matter. Alerts per rule, precision, true alerts found, minutes per true case, backlog and undetermined rate. Six numbers per rule, quarterly, with the baseline alongside them. That report is worth more than a hundred slides about model sophistication.

Watch for the shift you cannot see. A tuning program that cuts noise can quietly cut coverage. If your precision rises but your true-alert count falls for several cycles in a row, the rules are drifting toward under-detection, and you should pull a fresh sample of transactions that did not alert to test that hypothesis. Few programs do this, and it is the single strongest evidence you can hold that tuning has not compromised effectiveness.