How to Route Each Task to the Cheapest Model That Passes

Before you pick models, decide what counts as a task type. A task type is a group of requests that share the same definition of quality and the same cost of failure. A classification error sends a customer to the wrong queue. An inaccurate summary is harder to catch because it reads as credible. Combining both under one threshold lets strong results from the easy task hide a serious regression in the other Braintrust on model routing.

Pull the task types from production traces, not from a whiteboard. Logs already contain prompt templates, endpoints and token usage, and they show you which request shapes actually occur and in what volume Braintrust on model routing. A typical product ends up with four to eight types. For example:

Write the task type label into your request logs so every production call carries it. Then any future answer can be attributed to a type without guesswork.

You need enough tasks per type to trust a pass rate, and the mix must resemble what you will actually route.

Three rules keep the sample honest. First, sample per task type, not globally, because a rare type with a high failure cost deserves its own rows. Second, ignore the temptation to pick only interesting or hard examples: your pass rate must reflect the traffic the router will see. Third, hold the sample fixed: the cheap and strong models must answer the identical tasks in the identical wording.

On size, there is no universal number that makes a sample representative, which is the caveat any careful measurement template states the WIQ baseline template. A practical starting point is several hundred tasks spread across your types, with a floor per type (50 is a reasonable minimum) so a single unusual answer does not swing the rate. If one type carries most of your volume or most of your risk, sample it more heavily.

For evaluation done at agent scale, the numbers get much larger: quality comparisons between agent variants can need thousands of trajectories per arm to separate a real lift from model randomness, especially when the metric is subjective like helpfulness Maxim AI on A/B testing for agents. You are not running a significance test here, you are setting a routing rule with a fallback, so a few hundred well-chosen tasks per heavy type is enough to start, and the fallback absorbs the errors your sample missed.

Store the sample in a versioned file. Every task gets an ID, the task type, the input, and any context the request would normally have. You will rerun this file later, so treat it as a test suite, not a one-off spreadsheet.

A routing decision needs a binary outcome. Graders convert model output into pass or fail. Match the grader to the task type:

• Classification and extraction: exact match or schema validation. The label is right or it is not; the JSON has all fields and valid types or it does not. This is deterministic code, cheap and fast.
• Grounded answers (FAQ, policy): check that every claim maps to a retrieved passage. A simple version checks that quoted strings exist in the source document; a stricter version uses a second model as a judge with the source in the prompt.
• Drafting (tickets, emails): a rubric judge. Score four to six named criteria (correct facts, no invented commitments, tone, completeness, length) and pass only if every required criterion holds. Rubric language beats vibes: "mentions the refund window and cites the policy section" is gradeable, "good tone" is not.
• Summaries: check presence of key items, for example the decision, the owner, and the next step, detected by keyword or a judge prompt.

Two guardrails keep the graders trustworthy. First, hand-label a slice of the sample yourself (say 50 answers) and compare your labels to the grader's. If they disagree often, fix the grader before you trust any pass rate. Second, keep the judge prompt fixed for the whole run; a judge that drifts makes the comparison meaningless.

This is where the agent guidance about mixing deterministic software with models applies: use code where the check is mechanical, and reserve model judges for the parts that need reading comprehension Keith Cowing's rules for building AI agents that work.

Now run the sample twice, once per model, with the same prompt template, the same retrieved context and the same output constraints. Record for every task: task ID, task type, model, output, pass or fail from the grader, input tokens, output tokens, and any retry that happened.

Then compute two numbers per model per task type:

pass_rate      = passed tasks / tasks run
cost_per_task  = sum(all model spend on those tasks) / tasks run
cost_per_pass  = sum(all model spend on those tasks) / passed tasks

The third figure is the one that drives routing. It converts "cheap per token" into "cheap per useful answer." A model that passes half the tasks at half the price has the same cost per pass as the strong model, with none of the reliability. A model that passes most tasks at a tenth of the price wins clearly.

A worked example makes the comparison concrete. Suppose the billing-ticket draft type has 200 sampled tasks. The strong model passes 190 (95 percent) at an average of 4 cents per task. The cheap model passes 168 (84 percent) at 0.4 cents per task. Cost per pass is 200 times 0.04 divided by 190, about 4.2 cents for the strong model, and 200 times 0.004 divided by 168, about 0.48 cents for the cheap one. The cheap model is roughly nine times cheaper per passing draft. But 32 tasks failed, and each failure needs an escalation, so add the escalation cost before deciding.

If escalated cheap failures cost the same as a strong-model call (4 cents), the blended cost is 200 times 0.004 plus 32 times 0.04, which is 0.8 plus 1.28, or 2.08 cents in total, about 1.04 cents per pass. The cheap-plus-fallback path still wins on this type, at roughly a quarter of the strong-only cost. Run the same arithmetic on your own numbers; the decision rule is that a type goes cheap when its blended cost per pass, including escalations, is lower and the final pass rate is close enough to the strong model's.

The table below shows how such a decision sheet looks across several types, with invented but internally consistent numbers:

The refund decision row is the interesting one: the cheap model is much cheaper per attempt, but a 27-point gap in pass rate on a decision with real money attached is not "close." Route it strong. The extraction row sits in between, and that is where a fallback earns its keep.

The rule from the sample is simple to state and easy to over-tune. Keep three buckets:

1. Cheap always: the cheap pass rate is within a small tolerance of the strong model's and blended cost per pass is lower. Send it straight to the cheap model.
2. Cheap with escalation: the cheap pass rate is meaningfully lower but the blended cost per pass including escalations is still lower, and the failure mode is recoverable (a bad draft an agent would rewrite anyway). Send it cheap, grade it in production, and retry failures on the strong model.
3. Strong always: the gap is large, the failure is expensive or hard to detect, or the task carries regulatory or financial weight. Do not optimize here.

Cascades of this shape (cheap first, escalate on failure) can cut blended cost substantially when most traffic is resolved by the cheap tier, but the saving depends entirely on how much traffic the cheap tier actually clears TrueFoundry on cost and quality aware routing. That is why you need the pass rate per type rather than one global number.

A minimal router in code looks like this. The task type comes from your log label; the pass check runs only when the cheap model answers.

ROUTES = {
    "intent_classification": "cheap",
    "faq_answer": "cheap",
    "data_extraction": "cheap_then_strong",
    "ticket_draft": "cheap_then_strong",
    "refund_decision": "strong",
}

def answer(task):
    route = ROUTES[task.type]
    if route == "cheap":
        return call(CHEAP, task)
    if route == "strong":
        return call(STRONG, task)
    out = call(CHEAP, task)
    if grader(task.type, task, out).passed:
        return out            # log cost and the pass
    return call(STRONG, task)  # log the escalation

Log the route, the model, the tokens, and the grader result for every call. Without that log you cannot tell whether the cheap tier is still earning its place.

Two operational notes. First, do not route on prompt length or keyword matching as a substitute for the measured rule; those signals fire before any answer exists and cannot tell you whether the response will be good Braintrust on model routing. If you add a classifier later, it should be because the task label is genuinely unknown at request time, not because the label is inconvenient.

Second, routing is not the same thing as failover. Fallback for provider errors (timeouts, rate limits, server errors) is a reliability layer that belongs on every route regardless of quality GMI Cloud on model routing. The quality fallback in the code above answers a different question: was the cheap answer good enough?

A routing table is a hypothesis about the future, and assumptions about traffic and model behavior do not stay true on their own. Business planning guidance makes the same point about any documented assumption: record it, revisit it regularly, and adjust when the world moves, because an assumption that held six months ago may not hold now John Olivant on business planning assumptions. A plan you never revisit is a rigid plan, and rigid plans waste budget and trust John Olivant on business planning assumptions.

For routing, "revisit" means three triggers:

• Traffic mix change. A new feature or campaign shifts the share of each task type. If a cheap type grows from 10 percent to 40 percent of volume, re-sample it.
• Escalation rate drift. Watch the share of cheap answers that fail the grader in production. If it climbs, the cheap model is no longer clearing the type. Set a kill threshold before launch, for example escalate to strong for a type if the live failure rate exceeds its sampled rate by a set margin for a week.
• Rule change. If you change prompts, retrieval, or output formats, the old pass rates no longer apply. Rerun the sample.

There is a discipline that keeps this from drifting into rationalization: decide your threshold before you look at results. The clearest template for that comes from pilot design. First, a primary metric with a specific go threshold. Second, guardrail metrics that must not regress beyond a set tolerance. Third, qualitative signals recorded but not allowed to override the numbers. And crucially, if you define success after seeing the data, you are rationalizing, not evaluating the Institute of AI PM pilot template. The same structure gives you a defensible tolerance for "close enough" on pass rate, and a guardrail such as "no more than X escalations per 100 tasks."

One more practical tip from pilot guidance: document the analysis before you run it, including the smallest sample you will accept and how you will treat missing data the Institute of AI PM pilot template. A routing sheet that says "10-point tolerance, 50 tasks per type minimum, blended cost must beat strong-only by a stated margin" survives a reorg. A sheet that says "the cheap one looked fine" does not.

If your team is building this inside a product and wants one API key across cheap and strong models rather than stitching vendors together, Quanta Inference is the natural place to run the sample and the router: text, image and embedding models behind a single OpenAI-compatible endpoint. The measurement work is yours; the plumbing need not be. When you want a specialist to review the routing design and the graders with you, a free 30-minute discovery call covers your goals and a tailored recommendation (https://quantaintelligence.ai/calendar).