Queue Design for Fraud Review: What a Human Needs Beside a Score

Score is a probability, not a dollar amount. Two cases with identical scores can carry very different exposure: a high-value order, a large transfer, a high-lifetime-value account. Sorting by score alone means reviewers spend their scarce minutes on small cases while big ones age.

Expected loss is straightforward: estimated probability of fraud times the value at risk, adjusted for what you can still recover if the case goes bad.

$$\text{expected loss} = \text{score} \times \text{exposure} \times (1 - \text{recovery rate})$$

Worked example. Case A scores 0.85 on a 40 dollar order with recovery rate 0 (digital delivery). Expected loss: $0.85 \times 40 \times 1 = 34$ dollars. Case B scores 0.40 on a 2,000 dollar order with recovery rate 0.5 (physical goods, hold before shipment). Expected loss: $0.40 \times 2000 \times 0.5 = 400$ dollars. Case B is worth roughly twelve Case As. A queue sorted by score puts Case A first; a queue sorted by expected loss puts Case B first and uses the reviewer's minute where it protects the most money.

A comparison table makes the trade-off concrete:

The insight the first search result usually misses: the queue order is a business decision, not a model output. The model gives you a probability; the queue gives you a policy about which mistakes you would rather make. Document that policy next to the routing code, because when a disputed case reaches an auditor the sort order is part of the argument.

There is a second-order effect worth naming. Sorting by expected loss tends to shrink the queue without changing the model. Reviewers clear the expensive cases first, and the tail of tiny ambiguous cases can often be auto-approved with a lower confidence bar because the downside is bounded. That is a far cheaper fix than hiring.

A reviewer's job is not to interpret the score. It is to confirm or contradict it. That takes four blocks of information, each on the same screen, without a tab switch.

1. The top contributing signals, with raw values and direction. Not "model says 0.82" but "shipping address first seen on this order (risk up), email domain age 4 days (risk up), 14 prior orders completed with no disputes (risk down)." Show the raw value beside each so the reviewer can judge whether the signal is real or a data artifact. This mirrors how device intelligence is presented in practice: no single attribute proves fraud, and the value comes from reading conflicts between signals and relationships across accounts SEON.

2. The customer's recent history. Last five events, disputes, prior reviews and their outcomes. A case with no context invites a reviewer to over-weight the one signal they can see. Sardine describes the inspect step as pulling device, identity, history and documents together, which is the right mental model even if your screen shows fewer blocks Sardine.

3. Device, session and counterparty facts. Device hash seen before or new, how many accounts share it, IP and time zone versus declared location, VPN or proxy indicators, and whether the counterparty (merchant, payee, shipping address) has appeared in other cases. Device fingerprinting combines hardware, browser and network attributes into a persistent hash that survives cookie clearing, which is exactly the context a reviewer needs to see whether an event looks like the customer's normal Plaid. Cross-account link analysis, connecting profiles that share a hardware fingerprint, is what turns a single suspicious login into a pattern Sumsub. Note the honest limitation the same source states: no single attribute is unique to one device, so the identifier reflects the strength of the combination rather than a guaranteed one-to-one match, and the reviewer should treat a match as evidence, not proof.

4. The exact policy clause at stake. The rule identifier, the version, and the clause text. If the reviewer approves, which rule permits it? If they block, which rule requires it? Without this, decisions drift and root-cause analysis becomes guesswork.

A minimal case payload looks like this:

{
  "case_id": "c-88213",
  "score": 0.82,
  "exposure_usd": 412.00,
  "expected_loss_usd": 337.84,
  "signals": [
    {"name": "address_first_seen", "raw": true, "direction": "up"},
    {"name": "email_domain_age_days", "raw": 4, "direction": "up"},
    {"name": "completed_orders", "raw": 14, "direction": "down"}
  ],
  "history": {"last_5_events": "ref", "disputes": 0, "prior_reviews": 1},
  "device": {"hash": "d-9f31", "accounts_on_hash": 3, "vpn": false},
  "policy_clause": "POL-114 v7: new shipping address over 250 USD requires review"
}

The rule for the screen: if a reviewer needs a second tab to decide the case, the payload is incomplete and handling time will show it. Measure that directly by timing twenty cases before and after a screen change; you should see the median move, and if it does not the change was cosmetic.

One more design point. Signal values arrive from many sources on different clocks. Stamp each block with when it was read, so a reviewer is not comparing a device fact from this session with a history snapshot from three days ago and drawing the wrong conclusion.

Every decision should produce three things: the outcome (approve, block, escalate), a reason code, and an optional free-text note. Reason codes do three jobs.

First, they make reviewer behavior measurable. Count codes per reviewer and per week. A reviewer who marks "velocity" on 60 percent of blocks while colleagues mark it on 10 percent is either seeing something others miss or applying a different mental model. Second, they let you detect guideline drift: when the distribution of codes shifts sharply, a rule or guideline has probably changed without anyone noticing. Third, they become labels. Sardine frames reviewed cases as high-quality labels that train and tune the models behind them, precisely because a human decision on a genuinely hard case carries information the automatic system lacked Sardine. A review queue that produces decisions without structured reason codes throws that information away.

Keep the taxonomy short. Six to ten codes covering the signals that actually drive blocks will be used consistently; forty will not. Pair each code with the policy clause it maps to, so a block reason can be traced from decision to evidence to rule.

The labeling discipline matters beyond model training. Research on fraud detection repeatedly identifies the scarcity of labeled data as a core constraint on the field, which is why deliberate capture of human decisions is worth the small friction it adds Fraud Review Detection. The same paper notes that even expert human judges struggle to tell a fraudulent item from a genuine one in some domains, which is a useful humility check: the reason code does not have to be right every time, it has to be recorded so the error is visible later.

There is a cheap way to keep codes honest. Each month, sample thirty blocked cases and thirty approved cases, check them against later outcomes such as disputes or clean settlement, and report the mismatch rate per code. A code whose mismatch rate climbs is either mis-defined or mapping to a signal that has gone stale.

Inter-analyst agreement is the single most useful quality metric for a review queue, and it is almost always missing. The measurement is simple: sample cases and have two reviewers decide independently, then compute the share of cases where they agree. Low agreement on similar cases means the guidelines or the evidence are unclear, and the labels coming back to the model are noisy Sardine.

Here is the decision rule that turns agreement into threshold work:

That last row is the most expensive mistake in fraud operations. A rising queue tempts managers to hire, when the cause is usually upstream: thresholds and features that punt cases automation could decide. A review queue should be a precision tool for hard cases, not a bulk decisioning engine Sardine. When a queue grows because one rule started firing far more often, fixing that rule can drain the queue faster than any staffing change.

When you do retune, change one threshold at a time and watch three numbers: review rate, agreement, and decision accuracy against later outcomes. If accuracy holds and the review rate falls, you have found real capacity. If accuracy falls, you have moved the line past what the model can support. Keep the sample size honest: twenty paired decisions will tell you about a badly ambiguous code, but it will not measure a small accuracy shift, so do not over-read a single week.

A final disagreement signal is the escalation rate. If reviewers escalate to a specialist at wildly different rates for similar cases, the escalation bar is undefined. Write it down as a policy clause and it becomes testable.

Agentic tooling can assemble the evidence pack, pull linked accounts from a graph, and draft the rationale, which shrinks handling time and improves consistency across reviewers. A vendor walkthrough of this workflow is explicit that the value is in connecting evidence, reasoning across connected data, and generating explainable findings, not in replacing the judgment call, and that the systems flag the gap where a score arrives without enough explanation to act on TigerGraph. The broader case for agents in fraud work rests on the same premise: they process diverse sources such as transaction records and user behavior to give a fuller picture, and the persistent problem they address is that static rules struggle to keep pace LeewayHertz.

Two cautions. First, AI does not remove the need for labeled outcomes. The Association of Certified Fraud Examiners estimate, cited in that overview, that organizations lose about 5 percent of annual revenue to fraud, is a reminder of the stakes; the value of automation still depends on the structured data returning from human decisions LeewayHertz. Second, adversaries adapt. Research on collaborative LLM agents simulating financial fraud finds that malicious agents can adapt to environmental interventions such as content warnings and monitoring, which means your thresholds and review guidelines face an opponent that responds to them When AI Agents Collude Online.

The human must own three things regardless of automation: the final outcome on ambiguous cases, the reason code, and the policy clause cited. Everything else is fair game for tooling.

1. Measure arrival rate, median handling time and available reviewer minutes for one week.
2. Compute capacity with the formula above and compare to arrivals.
3. Add exposure and recovery to your routing so the queue sorts by expected loss.
4. Build the one-screen pack with the four blocks, and time a reviewer using it before and after.
5. Ship six to ten reason codes mapped to policy clauses.
6. Start a weekly agreement sample of twenty cases and use the table above to decide what to tune.

If you need the model switchboard behind this step (score, device signals, document checks, text classification through a single compatible API), Quanta Inference gives you one key and prepaid access across text, image, video, audio and embedding models: https://quantaintelligence.ai/inference. If the bigger job is wiring the evidence pack and routing into your own case tooling, describe what you need at https://quantaintelligence.ai/ and a specialist will build it with you.