Choosing a Model Per Task from Pass Rates and Token Counts

Capture, at request time, the metadata that lets you reconstruct a model-by-task picture later.

Tag at the call site rather than parsing logs afterward, because post-hoc parsing reconstructs context the application has already discarded, and a routed request shows up in raw logs as an opaque model call with only a provider, model and token count (Braintrust).

Take token numbers from the provider's usage object. On OpenAI endpoints, Chat Completions reports prompt_tokens, completion_tokens and total_tokens, while the Responses API reports input_tokens, output_tokens and total_tokens, and reasoning tokens are billed as output even though they never appear in the answer (OpenAI Help Center). A reasoning model can look cheap on visible output and expensive on the invoice.

import uuid

def call_model(client, model, task_type, prompt, max_out=800):
    resp = client.responses.create(model=model, input=prompt, max_output_tokens=max_out)
    return {
        "request_id": str(uuid.uuid4()),
        "task_type": task_type,
        "model": model,
        "input_tokens": resp.usage.input_tokens,
        "output_tokens": resp.usage.output_tokens,
        "passed": None,   # set by the checker, inline or in a batch job
    }

Write null for passed unless your checker runs inline. For schema-validated output the check can run in the same request. For anything a human reviews, backfill in a batch. The row is cheap; the gap is expensive. Also record the prompt template version next to the model, since a template change moves input tokens across every request in a feature.

A pass rate is only as good as the check behind it. You want a check that returns true or false without an argument. Three types cover most production work.

• Executable checks. The output runs: a test suite passes, a query returns rows, an artifact builds. Prefer this wherever the task produces something executable.
• Schema and constraint checks. The output parses against a JSON Schema, required fields are present, enum values are allowed, cited identifiers exist in your database. Cheap to run, catches many silent failures.
• Sampled human labels. A person marks a random sample of about 100 outputs per task type per week. The fallback for prose and tone-sensitive replies.

Bound the third type deliberately. Label a random sample, not a convenient one, and label the same sample shape for every model you compare. If you label model A in the morning and model B after a long day, your pass rates encode your fatigue.

For multi-step tasks, judge each step. Decomposition exists so a failure on step three is attributable to step three and can be rerun without replaying everything, which gives you measurable evaluations at each step rather than only at the end (Brightlume). A plan step and an execute step often want different models, so log them as separate task types.

The textbook Wald interval collapses at 0 and 100 percent and misbehaves on small samples. The Wilson score interval behaves correctly across the range and is a few lines of code.

import math

def wilson(passes, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = passes / n
    denom = 1 + z*z/n
    centre = (p + z*z/(2*n)) / denom
    half = (z * math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / denom
    return (max(0.0, centre-half), min(1.0, centre+half))

Now apply it to four model and task pairs on the same invoice_extract task.

B never beat A: its interval sits almost entirely below A's point estimate, and 40 samples cannot argue otherwise. The 5 point gap is noise. D's perfect record means nothing yet, because 12 clean runs give a lower bound under A's point estimate. C is genuinely better and tight enough to prove it.

Set a minimum sample size per pair before you allow a comparison. A working figure is 200 tries per pair, with 30 or more recorded failures. Below that, print "insufficient" rather than a number, because a dashboard that shows a rate for twelve samples teaches people to trust twelve samples. Treat the interval as a floor on your uncertainty, too: retries of the same input and documents from the same customer cluster, so true uncertainty is wider than the arithmetic suggests.

Token counts are input to the number you steer by. The arithmetic has three parts: cost per attempt, expected attempts per success, and the cost of the check itself.

cost_per_attempt = (input_tokens/1e6) * input_rate + (output_tokens/1e6) * output_rate
cost_per_success = total_cost_for_pair / count_of_passed_tasks

Use measured attempts per success from retry_of chains rather than 1/pass_rate, because failures correlate: a hard document stays hard.

Work the invoice example with plausible rates. Suppose the small model costs 0.30 dollars per million input and 1.20 per million output, the frontier model costs 5 and 25, and both consume 1,800 input and 400 output tokens per attempt.

• Small model: about 0.00054 dollars per attempt. Over 900 attempts that is about 0.486 dollars for 744 successes, so about 0.000653 dollars per success.
• Frontier model: about 0.019 dollars per attempt. Over 900 attempts that is about 17.10 dollars for 890 successes, so about 0.01922 dollars per success.

The frontier model costs about 29 times more per successful task. If a failed extraction triggers a manual review costing well under 3 cents, the small model wins outright. If the failure is a rejected invoice and a support ticket, the frontier model wins.

Correct twice before trusting the figure. Budget for overhead: retries, system prompts and context often warrant a 1.7 to 2.0 times multiplier on base estimates (iternal.ai). And check tokenization, because a lower price per million does not guarantee lower total cost, since models tokenize the same text differently and produce different amounts of output and reasoning (OpenAI Help Center).

You now have two coordinates per model and task pair: a pass rate with an interval, and a cost per successful task. Turn them into a rule anyone can apply without a meeting.

Set a quality floor per task from what the downstream consumer tolerates, not from what the best model achieves. A code_fix task checked by a test suite might need 95 percent because a broken merge is expensive; a tag_suggestion task might accept 85 percent because a human reviews every tag.

The rule: among models whose lower confidence bound clears the floor, choose the lowest cost per successful task. If none clears the floor, the task is not ready to automate. Fix the checker, split the task, or keep a human in the loop. The rule refuses to switch on a point estimate and refuses to compare a well-measured model against a thinly measured one.

When two models clear the floor and the cost difference is small, prefer the tighter interval over the higher point estimate. Recheck monthly rather than continuously, on a stable task taxonomy and checker. Continuous reshuffling on partial data produces unstable routes and makes a cost regression impossible to attribute.

One case the rule handles badly: task types whose difficulty varies wildly inside them. If support_reply contains both "where is my order" and "explain this charge", split it into two task types and route separately. Granularity detailed enough to execute but not so fine that the system becomes brittle is the right target (promptessor). Routing cannot fix a taxonomy that is too coarse.

Define task types. ticket_intent classifies the request, ticket_reply_draft writes the first draft, ticket_escalation_check decides whether a human takes over. Three fixed labels, no free-text categories.

Instrument. Every call logs task type, model, both token counts, cost, and passed. ticket_intent checks against a closed intent list and sets passed inline. The other two backfill from a weekly sample of 100 labelled outputs, plus a rule that an escalation check is wrong if a human later reclassified the ticket.

Measure. After three weeks, ticket_intent has thousands of tries with tight intervals around 91 percent and 94 percent, the frontier model costing an order of magnitude more per success. ticket_reply_draft has 220 labelled samples per model and a wide interval for the small model, which labelers mark as too terse. ticket_escalation_check has a 98 percent floor because a missed escalation is a service failure.

Route. The rule picks the small model for ticket_intent. Neither model clears the floor for ticket_reply_draft at a confidence people trust, so drafts stay on the frontier model while the team collects labels and adds a terse-is-a-fail rule. Only the frontier model's lower bound clears 98 percent on escalation, so it keeps the frontier model despite the cost.

The insight: good routing produces a mixed fleet, not a single winner. Anyone can audit each choice by reading the log.

When the fleet spans several providers and you want token and cost figures in one shape, Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key, prepaid and priced in US dollars. The routing rule above does not change; only the billing plumbing gets simpler.