How to Set the Cheap Model's Cleanliness Bar for Bank Row Routing

Before you label a single row, fix three numbers. Without them, no threshold is defensible.

1. Base rate of dirty rows, d. The share of incoming rows that genuinely need correction. In a well-parsed ingestion pipeline this is often small, which is exactly why the cheap model's mistakes hide.
2. Tolerated miss rate, m. The share of all rows you are willing to let through as clean when they are dirty. This is a business decision, not a model decision. If one silent miss per 10,000 rows is acceptable, m = 1/10,000 = 0.0001.
3. Cost ratio, r. How many times more expensive one escalated row is than one row handled by the cheap model alone. Include the bigger model's tokens plus any queueing or latency penalty.

The decision rule falls out of these:

minimum escalation rate = (false_clean_rate × d) / m

Notice the structure. The numerator is the number of silent misses per row of traffic if you escalated nothing. The denominator is what you will tolerate. Their ratio is the factor by which escalation must exceed the base rate implied by tolerance alone. If the false-clean rate is 1 in 500 and you can tolerate one miss per 10,000 rows, you need to escalate at least 20 times more often than the tolerance floor suggests.

Here is a worked example with the numbers kept simple so you can substitute your own. Suppose d = 0.02 (two dirty rows per hundred), the measured false-clean rate is 0.002 (1 in 500), and m = 0.0001 (1 in 10,000). Then required escalation = (0.002 × 0.02) / 0.0001 = 0.0004 / 0.0001 = 4. Interpreted plainly: you must escalate 0.04% of all traffic, which is 2% of the rows the cheap model would have cleared, to hold the miss budget. At 1 million rows a month that is 400 escalations, and you can check the cost against r in the next section.

The false-clean rate is an empirical quantity, so sampling design drives everything. Three choices decide whether your number is real.

Sample from real traffic, not from a curated set. Rows you have already flagged as suspicious are not representative. Draw randomly from the rows the cheap model actually labels clean in production. A stratified sample that overwrites the clean-label stratum will systematically inflate your error estimate.

Size the sample for the rate you expect. To detect a 1 in 500 event with reasonable precision you need enough clean-labeled rows that expected misses are comfortably above zero. If the cheap model clears 90% of 5,000 sampled rows, you have 4,500 clean-labeled rows and expect about 9 misses at a 1 in 500 rate. That gives you a confidence interval wide enough to be honest about it, and it is why a sample in the low thousands is a floor, not a target. Treat the estimate as an upper and lower bound and set the threshold on the pessimistic end.

Use two independent labelers on every sampled row. Dirty is a judgment: a duplicate reference, a merged payee, an amount that does not net to the statement total. Agreement between two labelers is your evidence that the label is reproducible. Where they disagree, resolve it explicitly and record the rule.

The measurement itself is a confusion matrix on the clean label, which is the standard way to read a classifier's error structure (Pico's explanation of the false positive rate). Keep the four cells separate and never collapse them:

One methodological warning from the survey literature: a single false-clean rate averaged over all rows can mask very different rates by segment. Deposit rows, card settlement rows, and FX conversion rows may each have their own rate, and routing on the global average under-protects the worst segment. Measure per segment and set the threshold on the highest-risk one you cannot route separately.

A threshold only makes sense on a score. Bank row routing usually has one, whether it is the cheap model's stated confidence in a structured verdict, token-level margin, or a deterministic validator score. The calibration step is to find the score below which escalation fires.

The procedure, adapted from the calibration-first approach in the cascade literature (UCCI):

1. Score every row in your labeled sample with the cheap model and record the score alongside the label.
2. Fit a monotone mapping from score to the probability that the row is actually dirty. Isotonic regression is the usual tool, and it is what turns a raw confidence signal into a usable risk estimate.
3. Sweep candidate thresholds. For each, compute escalation rate on the sample and expected silent misses using the fitted probabilities.
4. Pick the lowest threshold whose expected vulnerable miss rate is at or below m. The lowest, not the lowest-looking one, because every point higher spends money for safety you already have.

If your platform returns schema-validated JSON, you can also gate on structural signals, which are cheaper and more stable than verbalized confidence. Models can produce typed output that validates against a schema, either through constrained decoding or through a validation-and-retry loop (Snowflake's structured output documentation, Databricks on structured outputs for batch and agent workflows). A row where the model had to retry twice before the amount, currency, and reference fields validated is a good escalation candidate regardless of its stated confidence.

The failure mode to avoid is the one the cascade literature calls out directly: confidence signals are often miscalibrated and sensitive to prompt wording, so a threshold tuned on one workload frequently fails on another (UCCI). Verbalized self-assessment is the weakest signal of the three common types, while probe-based methods that read internal activations are the most accurate but require white-box access (Concentrate on confidence-threshold escalation). If you are on a closed API, calibrate the score you have rather than trusting it.

A decision rule for when to escalate above the threshold: escalate every row whose calibrated dirty probability exceeds m divided by d, when escalation is costless. When escalation costs r, add the constraint that escalation should only fire when its expected cost is lower than the expected cost of a silent miss. In practice this means the threshold sits where the marginal reduction in silent misses per escalated row equals m times the number of rows per escalation. Work through it once in a spreadsheet and the number becomes obvious.

Take a mid-size payments processor with 2 million bank rows a month and a measured staff-verified data quality process. The numbers below are illustrative so you can see the arithmetic; substitute your own measurements.

Required escalation rate = (0.004 × 0.015) / 0.00005 = 0.00006 / 0.00005 = 1.2. That is 120% of the base surveillance level, or about 1.2% of total rows, meaning roughly 24,000 escalations a month. At r = 12, the extra cost relative to escalating nothing is what the threshold formula asks you to compare against the expected cost of the silent misses you are preventing. Do that comparison explicitly and put both numbers in the review document. The argument that survives a risk committee is not "the model is good"; it is "we escalate 1.2% of rows to hold silent misses under one in 20,000, and here is the monthly cost."

The worked example also shows why the false-clean rate is the one number that determines the escalation rate. Halving it to 0.002 halves the escalations needed to hold the same miss budget. That is the lever: every point of false-clean rate you remove from the cheap model is twice as valuable as a point of threshold tightening, because it buys budget without spending tokens.

For teams running a cascade like this across several document or transaction types, keeping all the model calls behind one OpenAI-compatible endpoint removes a class of integration work: you swap the cheap and large models by changing a model name, not by rewriting clients. Quanta Inference exposes text, vision and embedding models through a single prepaid API key, which is enough to run the labeling study and the calibrated router in the same code path.

The false-clean rate is not a constant. It moves when the merchant mix shifts, when a new bank statement format arrives, when the prompt is edited, or when the cheap model itself is swapped. Each of those is a reason to re-run the calibration, and the cheapest version is a small continuous audit.

Set aside a random slice of production rows every week, label them, and track three numbers: the false-clean rate overall, the false-clean rate per segment, and the share of escalations that turn out to have been clean. If the last number rises, your threshold is too aggressive and you are paying r to be safe you already are. If the first number rises while escalation volume is flat, you have a silent drift problem and the bar must move.

Two guardrails keep the audit from becoming theater. First, keep the audit sample physically out of any prompt or fine-tuning set; leakage inflates performance and deflates the false-clean rate exactly where it matters. Second, log the raw model verdict, the structured output, and the routing decision for every row, so a later investigation can reconstruct why a dirty row passed. Reconciliation practice from accounting makes the same point: unmatched lines are the signal, and the audit trail is what lets you resolve them (Microsoft's bank account reconciliation guide).

Where the cheap model's verdict has a confidence threshold already exposed by the vendor, treat that number as a hypothesis to test, not a fact. The confidence-token line of research shows that models can be trained to express confidence reliably and that this beats both verbalized confidence and raw token probabilities for downstream routing and rejection decisions (ICML poster: Learning to Route LLMs with Confidence Tokens, OpenReview version of the same paper). If you can choose between two cheap models, prefer the one whose confidence has been validated on your own rows.

Finally, treat the tolerated miss rate as a contract with the business. Write it down, name who owns it, and revisit it when the cost of a miss changes. The threshold is a consequence of that contract plus the measured false-clean rate, and nothing else. When the business wants fewer misses, it is buying escalations; show it the arithmetic and let it choose the number.