How to Choose the Number of Score Bands for a Tiered Workflow

An error budget is the amount of unreliability a system is allowed to produce over a period. In reliability work, a service with a 99.9 percent uptime objective receives an error budget that permits about 43 minutes of downtime per month error budget practice. The number is set from what the business can absorb, then spent deliberately.

For a tiered classification or routing workflow, the same logic applies. The "downtime" is the fraction of items whose action was wrong: a low-risk item sent to a costly reviewer, or a high-risk item approved automatically. Pick the total budget from the cost side, not the model side. If a wrong high-risk decision costs 40 times a wrong low-risk decision, a flat budget across the whole population is the wrong shape.

A small worked framing helps. Suppose you process 100,000 items a month and you are willing to tolerate 5 percent of them receiving a wrong action, whatever the band. That is 5,000 items a month. That total is the number you are splitting. It is not a target you hope to hit; it is a ceiling you track on a validation set and then in production.

The reason to fix the total first is that it forces the conversation about cost. A team that says "we want 99 percent accuracy" has not yet made a business decision. A team that says "we can absorb 5,000 wrong actions a month, but only 1,000 of them may be in the high-cost band" has. The second statement can be tested.

Once the total is fixed, allocate shares by cost. The band where a wrong decision triggers the largest downstream cost, whether that is a payout, a compliance exposure or a customer escalation, should receive the smallest share. The band where the wrong decision is a minor inconvenience gets the largest share. This is the inverse of what intuition suggests: teams often spend the most care on the middle band because it feels ambiguous.

Inverting cost into share means ranking the bands by the cost of a wrong action and assigning shares that decline as cost rises. A practical default is to allocate roughly 20 percent of the budget to the highest-cost band, 30 percent to the middle, and 50 percent to the lowest-cost band. The exact split depends on your cost ratios, but the direction should always hold.

Here is how it looks with the 5 percent budget from the previous section.

The split is 20/30/50 of the total budget. The high-risk band carries only 1,000 wrong actions a month even though its per-item cost is enormous; the low-risk band absorbs 2,500 because each one is a rounding error. If you had split the budget evenly, the high-risk band would have been allowed 1,667 wrong actions, two-thirds more exposure than the cost profile justifies.

Two adjustments are worth knowing. First, if any single high-risk wrong action is catastrophic and irreversible, cap that band below its proportional share and treat any breach as a stop-the-line event rather than a budget spend. Second, if the middle band turns out to be cheap because the audit is automated, you can shift its share upward and lower the high-risk band further. The shares should follow measured cost, not the labels.

With shares in hand, the boundary problem becomes a search. You need thresholds t1 and t2 such that items scoring above t1 and below t2 land in the middle band, items above t2 land in the high-risk band, and the observed error rate in each band on a validation set is at or below that band's share.

The wrong way is to pick 0.7 and 0.9 because they look neat. The right way is to sort your validation set by score, then walk the score axis from the safe end until the error rate in the accumulated low-risk region rises to its ceiling, and put t1 there. Repeat for t2. The thresholds will rarely be round.

This mirrors how score tiers are set in other disciplines. In assessment tools, threshold levels map a candidate's calculated score to a reward band, and the tooling explicitly warns that the ranges should be tested with different answer combinations to confirm the reward behaves as intended threshold scoring guidance. In sales-engineering tools, the guidance is blunter: thresholds are a business decision, drawn where your win data and team capacity say they belong, and every tier should trigger a distinct action fit score thresholds. A tier that changes nothing is decoration.

Calibration comes before any of this. A threshold on a miscalibrated confidence score is a threshold on the wrong axis, and the recommended practice is to calibrate the score before thresholding it cascade optimal stopping analysis. Concretely: bin your validation scores, compare the mean predicted confidence in each bin to the observed accuracy in that bin, and fit a correction if they diverge. Only then measure error rates per band.

A short script captures the walk. It assumes a numeric risk score where higher means riskier, and it returns the thresholds that just satisfy each band's ceiling.

import numpy as np

def pick_thresholds(scores, wrong, budget_share, min_band_size=200):
    # scores, wrong are arrays aligned to validation items
    # budget_share is a list, low risk to high risk, summing to total
    order = np.argsort(scores)          # ascending risk
    s, w = scores[order], wrong[order]
    n = len(s)
    edges, start = [], 0
    remaining = 1.0
    for share in budget_share[:-1]:
        allowed = share * n
        seen_err, edge = 0.0, start
        for i in range(start, n):
            seen_err += w[i]
            edge = i
            if seen_err > allowed and (i - start) >= min_band_size:
                break
        edges.append(s[edge])
        start = edge + 1
    return edges

Run it on a validation set you have not trained on, then report the realized share per band, not the requested one. If a band comes out far under its share, you can move the boundary to buy accuracy elsewhere. If it comes out over, the boundary is wrong or the score is not informative enough and no threshold will fix it.

The number of bands is a cost-benefit question, and the research is unhelpful to optimism. A study that optimized model sequences and deferral thresholds across a pool of eight models found that two-model cascades improved mean test-set accuracy over the best single model by 2.1 to 8.2 percentage points, but allowing more than two models did not improve mean accuracy in 118 of 135 comparisons and added at most 0.43 percentage points depth of LLM cascades. The gains from extra depth depended on how well the score separated correct from incorrect answers for the current model versus later ones, and disappeared when scores also reflected query difficulty shared across models.

Translate that into a decision rule. Add a band only when you can name a distinct action with a distinct cost, and only when your validation set shows the new boundary improves the weighted cost, not just the accuracy of some subset. Three bands is usually the practical ceiling for a workflow with human review in the loop; four or more bands tend to collapse into "review" and "no review" the moment an operator looks at them. Analytics tooling guidance converges on the same range, recommending usually two to five tiers and warning that more tiers confuse the people reading the output score tier configuration.

There is also a measurement trap. Outcome metrics such as final task completion hide failures on the path, and production agents can finish a job through an unsafe or expensive route that never shows up in a single success rate agent evaluation framework. For a tiered workflow, measure per band: how many items landed in each band, what fraction of band decisions were wrong, and what the wrong ones cost. If a band's realized cost exceeds its share for two consecutive review periods, tighten that boundary or move the action up a tier.

Take a support operation that classifies inbound refund requests. The score is a calibrated risk estimate: higher means more likely to be an abusive or high-value claim. There are three actions: auto-approve under a small cap, approve with an automated audit check, or send to a human specialist. The cost ratios are 1, 8 and 40 as in the table above, and the total budget is 5 percent.

1. Calibrate the score against a labeled sample of 20,000 historic requests. Compare predicted risk to observed abuse rate per score bin and fit a correction.
2. Split the budget: 2.5 percent to auto-approve, 1.5 percent to audited approval, 1.0 percent to human review.
3. Walk the score axis from safest upward. At a score of 0.08, the cumulative wrong-approval rate in the auto band reaches 2.5 percent. That is t1.
4. Continue from 0.08. At a score of 0.31, the wrong-approval rate in the audited band reaches 1.5 percent. That is t2. Everything above 0.31 goes to the specialist.
5. Check the volume. If t2 sends 30,000 of 100,000 requests to specialists, the human queue is the binding constraint and the band structure is unaffordable regardless of its error rates. In that case widen the audited band and accept a higher share there, or add automation to the specialist queue.

Step 5 is the one teams skip. A boundary that satisfies the error budget but breaks the capacity plan is not a valid boundary. The two constraints must hold together.

This connects to a broader point about agent and model selection. A practical guide from OpenAI recommends building with the most capable model first to establish a performance baseline, then swapping in smaller models to see whether they still meet the accuracy target practical guide to building agents. The analogue for bands: establish the strictest band first, measure what it costs to be right, then relax each lower band until its error share is spent. You end up with a tier structure that is derived from cost, not from a model catalog.

When you are ready to run several model tiers behind one interface, Quanta Inference gives you text, image, video, audio and embedding models through one OpenAI-compatible API key, prepaid and priced in US dollars (https://quantaintelligence.ai/inference). That removes the plumbing question and leaves the band design, which is where the cost actually lives.

The failure modes repeat across teams. Watch for these.

• Silent thresholds. The boundary exists in a design doc but nothing in the routing system reads it. The ACC-style guidance on escalation is explicit that a threshold which triggers nothing is a documentation artifact, not a control escalation governance metrics.
• Approver fatigue. Boundaries set too low flood the expensive band with items that should have stayed cheap, and reviewers start ignoring the queue. The fix is to raise the boundary and accept a higher error share in the cheaper band, not to add reviewers.
• Bands with no distinct action. If three bands all end in "someone looks at it," you have one band and a queue.
• Uncalibrated scores. A model can produce fluent, confident output that is wrong, so token probability is a poor proxy for correctness until you calibrate it model routing and cascades.
• Ignoring correlated errors. If the cheap model and the expensive model fail on the same queries, escalation buys less than the score's apparent accuracy suggests, and no threshold recovers it cascade regret analysis.
• Budget shares that never move. Revisit shares when the cost of a wrong action changes. A new compliance rule that raises the high-risk cost should shrink its share the same week.