How to Route Bank Rows Between a Cheap and a Strong Model

Build the router in two stages. Stage one is free: deterministic rules over the row itself, no model call. Stage two is the paid cascade: small model first, escalation only when the small model's output is below a confidence threshold.

Stage one flags a row as ambiguous when any of these are true:

Rows that pass all checks go straight to the small model. Rows that fail any check either go straight to the large model, or go to the small model with the failure recorded so the threshold can catch them. Rows that pass the pre-check but whose small-model output comes back under the confidence threshold also escalate.

Bank reconciliation is fundamentally a matching problem between statement lines and internal ledger entries, and mismatches fall into a known set of causes: a transaction in the bank feed with no matching posting, a posting with no bank transaction, or a timing difference. Microsoft documents exactly these categories and the resolution for each in its bank reconciliation guidance, which makes a good basis for your stage-one rules. Statement parsing itself has a separate failure mode, so keep parsing bank statement files as its own step before classification.

A minimal version of the gate looks like this:

def needs_large_model(row, small_result, threshold=0.70):
    if not row.get("description") or row.get("amount") is None:
        return True
    if len(row["description"]) < 12:
        return True
    if row.get("amount_mismatch"):
        return True
    if small_result["confidence"] < threshold:
        return True
    return False

The threshold is the one number that decides the whole cost-quality trade-off. Too high and you escalate constantly, paying more than sending everything to the large model would have cost. Too low and the small model keeps handling rows it cannot do, quietly lowering quality with no error logged. Calibrate it on a sample of rows where the correct category is already known, find the point where the confidence signal best predicts the small model's own mistakes, and set the threshold there rather than from a guess or from a benchmark that does not resemble your export.

You have three classes of signal, and they differ a lot in reliability and in engineering effort.

Verbalized self-assessment ("I am not sure this is payroll") is the weakest, because models are not well calibrated at describing their own uncertainty in words. Log-probability based signals are better and cheap to collect: take the probability distribution over your allowed category set and use the top probability, or 1 minus it as an uncertainty score. A query where one category clearly dominates stays on the small model; a query with several plausible answers escalates. Training-based approaches, such as teaching a model to emit explicit confidence tokens, perform best on routing and rejection tasks according to the ICML poster on learning to route with confidence tokens, but they need a training loop and access to model internals, so they only work with open-weight models.

There is a real caveat you should design around. Confidence routing is task-dependent. A systematic evaluation across three model families, four languages and three task types found that confidence identified useful escalations on reading comprehension and topic classification (mean correction-detection AUC of 0.612 and 0.705), but on a knowledge and reasoning dataset the mean AUC was only 0.510 and routing gave no consistent improvement over random selection, because small models often stayed highly confident while wrong. The same study found the method can create language escalation gaps of up to 30.33 percentage points, which equal-quota routing removed with little aggregate accuracy cost. The full results are in Confidence-Based Small-to-Large LLM Routing.

The practical translation for a bank export: confidence works well for categorization and matching, which are closer to classification than to open-ended reasoning. Test it on your own rows before trusting it. If a subpopulation (say, non-English merchant descriptions) is systematically under-escalated, apply a quota so that each subpopulation escalates at a similar rate rather than letting a global threshold allocate compute unevenly.

One more signal worth considering is a verifier rather than a self-report. If your small model must output a structured object with a category, a confidence, and a cited rule, you can validate the object against a schema and reject anything that does not conform. Structured outputs are designed exactly for this: you define a JSON Schema and get typed data back instead of free text you have to parse, as described in Snowflake's structured output documentation. Databricks notes that batch structured generation with a response format is built for precisely this case of extracting fields reliably across millions of data points, and that constrained decoding limits which tokens can be returned at each step. See structured outputs for batch and agent workflows. AWS covers the same technique in more depth with the Outlines framework in this post on structured output from LLMs.

A schema-validated response plus a confidence field is a stronger gate than a single number from a chat completion, because a malformed answer is an automatic escalation rather than a silent failure.

Assume an export of 100,000 rows. The illustrative prices below are placeholders you should replace with your own contract rates: a small model at 0.10 USD per million input tokens and 0.40 per million output tokens, a large model at 3.00 and 15.00, and an average of 600 input and 120 output tokens per row with the shared instruction prompt cached.

Now add routing. Suppose the deterministic pre-check costs nothing per row, runs at about 1 millisecond of a modest compute instance, and you value that instance at 0.05 USD per hour. Across 100,000 rows at 1 ms each that is 100 seconds, or about 0.0014 USD. Effectively zero. The real routing overhead is the small model call you make on every row you do not pre-escalate, plus the large model call on the escalated share.

With a 12 percent escalation rate:

• Small model handles 88,000 rows: 100,000 x 0.000108 x 0.88 = 9.50 USD
• Large model handles 12,000 rows: 100,000 x 0.00360 x 0.12 = 43.20 USD
• Routing logic: about 0.0014 USD
• Total: about 52.70 USD, or 0.000527 USD per row

Against 360.00 USD for the all-large baseline, that is roughly a 85 percent reduction. The blended average sits much closer to the small model's price than the large model's, which is the whole point: the average cost per row is dominated by whichever model handles most rows.

This is the same structure as a contribution margin calculation, where every incremental dollar of variable cost has to be earned back. If your per-row economics are revenue minus variable cost, then reducing variable cost per row from 0.0036 to 0.00053 raises the contribution of each row by 0.0031 USD. On a 100,000-row month that is about 310 USD of margin, and the arithmetic scales linearly with volume. For how contribution margin and break-even work in general, the contribution margin formula and examples walk through the per-unit version, and Contribution Margin is King makes the case that it, not revenue, is the number that tells you whether a unit is worth producing.

The break-even point for routing is the escalation share at which the blended cost equals the all-large baseline. Solve for the escalation rate e:

Blended per row = (1 - e) x small + e x large + routing

For the numbers above, small is 0.000108 and large is 0.00360, so blending always lands below the all-large cost for any e below 1. That means routing in a two-model setup almost always beats a single large model on pure token cost, and the real question is whether it beats doing nothing at all on quality. Compare instead against the all-small baseline: routing costs more than all-small by exactly the escalated share times the price gap, and that is the premium you pay for accuracy on the hard rows.

The decision rule that follows:

• If your error cost per misclassified row is below about 0.0035 USD, send everything to the small model.
• If it is between roughly 0.0035 and 0.05 USD, route, and tune the threshold until escalation lands where the quality gain is concentrated.
• If it is above about 0.05 USD (tax-filing errors, fraud flags), send the flagged rows to the large model regardless of confidence.

That middle band is wide, which is why the threshold, not the model choice, is where most of the work is.

Work out your own break-even by writing down three numbers: small model cost per row, large model cost per row, and the cost of one error. Then set the escalation rate so that the marginal escalated row's expected error reduction is worth more than the price gap. If you would rather not run the arithmetic or hold contracts with three providers, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, prepaid and priced in US dollars, which makes it straightforward to price a small and a large model against the same export and swap between them per row.

Three implementation details decide whether the routing holds up in production.

First, batch. Rows from a bank export do not need real-time processing. Batch inference lets you group rows into one call with a shared prompt and amortize the instruction tokens across many rows. On the numbers above, input is about 87 percent of each row's token count, so caching or sharing a long categorization prompt is the single largest lever after routing itself. One bank that moved to LLM transaction categorization, ANNA, reported cutting LLM costs by 75 percent through offline prediction, better context use and prompt caching, as summarized in this LLMOps case study.

Second, enforce a schema. Every model call should return the same object shape so the router can read the confidence field and the category, and so a malformed answer is a validation failure rather than a parse error three steps later. The structured output pattern is worth reading before you write the prompt, because the retry behavior matters: a model that fails schema validation twice should be escalated rather than retried indefinitely.

Third, log the routing decision per row. Store the pre-check failures, the small model's confidence, whether it escalated, the final category, and whether a human later corrected it. Without the correction column you can never recalibrate the threshold, and without the escalation column you cannot see when a rule has started over-firing. Reconciliation work is exception-based by nature, so review only the rows that failed, not the whole export, as the ad spend reconciliation playbook argues: automate the match, then review exceptions.

A small worked example of the payoff of the audit trail: if your form says 12 percent escalation but you find that one generic merchant name ("SQ *") accounts for a third of escalations, tighten a rule for that merchant and your escalation rate falls, which lowers your blended cost without touching the model.

Escalation routing is not free of downside. It adds latency on escalated rows (two calls in sequence, or one if you pre-escalate from the cheap rule), adds a threshold you must calibrate and maintain, and adds a failure mode where the small model is confidently wrong. Three cases argue against it.

When the export is small. The setup, calibration and monitoring effort exceeds the token savings on a few thousand rows. Route only when monthly volume is high enough that the absolute saving covers the engineering time, and use the break-even table above to check.

When the task is open-ended reasoning rather than classification. The evidence that confidence predicts useful escalation was weak on knowledge and reasoning tasks, with AUC near chance. If your rows need multi-step judgment, either send them all to the large model or build a verifier that checks the output against the ledger rather than trusting self-reported confidence.

When the cheap rule fires so often that the small model rarely runs. If your pre-check flags 60 percent of rows as ambiguous, the cascade degenerates into a mostly-large pipeline with extra steps and extra latency. Measure the flag rate before you build: if it is above roughly 40 percent, fix the data quality or the rules first.

For teams where the routing logic itself is the hard part, or where a line of business needs the export classified end to end, a short scoping call with an AI specialist is the fastest way to size the threshold and the rules against your actual data. You can book a free 30-minute AI discovery call with your goals and get a tailored recommendation, no obligation.