A Cost Per Accepted Output: The Metric That Decides a Model Swap

You cannot count accepted outputs until you can say what acceptance means. A quality bar is a written, testable definition of a good output for one task: which fields must be present, which format is required, which claims must be grounded in provided context, which safety constraints apply. Without it, every candidate looks acceptable because nobody is scoring.

Two existing practices translate well. First, evaluation criteria paired with a dataset of inputs and expected behaviors, built for the specific prompt rather than borrowed from a generic benchmark (Seya). Second, a golden dataset, the fixed set of cases you run before and after every meaningful change, which is what makes a before/after comparison honest (Langfuse). A golden dataset also lets you expand coverage with synthetic cases when real ones are scarce (Evidently AI).

The Quality Gate pattern from software engineering is the model to copy. Thresholds tightened for pull requests catch regressions that a project-wide gate misses (SonarSource community). For a model swap, the equivalent is a gate that runs on a sample of live and golden traffic and is tightened deliberately: if the current model passes 92 percent and a candidate passes 86 percent, that gap is the signal, not a rounding error.

Keep the bar binary for the main metric. Scored rubrics (1 to 5 on tone, 1 to 5 on completeness) are useful for diagnosis, but cost per accepted output needs a pass or fail. A practical default: an output passes when every hard requirement is met and no hard requirement is violated. Soft preferences live in a separate score you review during failure analysis.

Two rules keep the bar from drifting. Version it, and record the version in every cost record. If you change the bar midway, the two halves of the data measure different things. When a bar change is unavoidable, re-score the current model's retained outputs under the new bar to establish a new baseline instead of comparing across versions.

The metric is only as good as the join between spend and acceptance. Attribution has to happen at request time: tag each call with a user, feature, deployment, prompt version, agent run and customer identifier, and use a proxy or SDK layer rather than reconstructing context from logs later (Braintrust). Those tags let you roll spend up by candidate and by feature, and join it to acceptance records from your judge or your reviewers.

Acceptance comes from one of three instruments, and they differ in trust. An automated check (schema validation, required fields, regex, retrieval grounding assertions) is cheap and stable. An LLM judge is flexible but needs calibration against human labels before you rely on it for pass rates. Human review is the ground truth and the most expensive, so use it on a stratified sample.

For model swaps, the cleanest setup is a two-tier gate. Rule checks run on everything, so retries and failures are caught and billed correctly. A judge or human review runs on a sample, sized large enough that a 5 point pass-rate difference is distinguishable from noise. Record the instrument and its version in the same record as the cost, so a change in the judge can be spotted as a step change in pass rate.

Do not let retries vanish. When a request fails and is retried, the failed attempt still cost money. If your logs only show the successful call, your numerator is smaller than your bill. The retry-storm failure mode is well documented: SDK retries with backoff log the final completion, and the intermediate tokens never surface (Braintrust). A cascade or fallback architecture has the same trap, because you pay for the first attempt and the fallback (Merge).

The formula is one division:

cost_per_accepted_output =
  (input_cost + output_cost + retry_cost + failed_call_cost + judge_cost)
  / accepted_outputs

Count judge cost in the numerator. If you spend money scoring candidates, that cost is part of delivering an accepted output, and it differs by candidate because a weaker model needs more or longer judged outputs. Some teams leave it out for simplicity, which is fine as long as every candidate is treated the same way and the omission is documented.

Worked example. Two candidates, same task, 10,000 requests.

• Candidate A: priced at 2.50 per million input tokens and 10.00 per million output tokens, with a request profile of 1,200 input and 350 output tokens. First-try pass rate 88 percent. Rejects are retried once at 1.6x tokens (prompt plus repair instructions plus prior output), and 60 percent of retries pass. Judge cost 0.20 per 1,000 scored outputs.
• Candidate B: priced at 0.90 per million input and 3.60 per million output, same profile. First-try pass rate 71 percent. Same retry policy; 65 percent of retries pass.

Do the arithmetic for 10,000 requests.

Candidate A: 8,800 accepted on the first try. 1,200 rejects become retries, and 720 pass. Total accepted 9,520. Token cost is 10,000 first attempts plus 1,200 retries, so 11,200 request-equivalents. Each request-equivalent is 1,200 input tokens at 2.50/M (0.003) plus 350 output at 10.00/M (0.0035), which is 0.0065. That gives 72.80 in model spend. Add judge cost of 0.20 per 1,000 scored outputs times 11,200 scored, which is 2.24. Total 75.04. Cost per accepted output is 75.04 / 9,520, about 0.0079.

Candidate B: 7,100 first-try accepted. 2,900 rejects, and 1,885 pass. Total accepted 8,985. Request-equivalents: 12,900. Each is 1,200 at 0.90/M (0.00108) plus 350 at 3.60/M (0.00126), which is 0.00234. Model spend 30.19. Judge cost 0.20 per 1,000 times 12,900 scored is 2.58. Total 32.77. Cost per accepted output is 32.77 / 8,985, about 0.0036.

Candidate B still wins on cost per accepted output, and by a wide margin. The cheaper token price here is not a trap. The trap appears in the next section, where the gap in pass rate is larger and the repair attempt is more expensive than the first call.

The point is not that cheap models always lose. It is that the pass-rate and retry terms can dominate. Change three things: Candidate B's first-try pass rate falls to 55 percent, its retries pass only 40 percent, and a retry costs 2.5x a first attempt because the repair prompt needs retrieved context and stricter instructions.

Candidate B then accepts 5,500 on the first try. 4,500 rejects, 1,800 pass, total 7,300 accepted. Request-equivalents in first-attempt terms: 10,000 plus 4,500 for the rejects at their first-attempt cost, plus 4,500 retries at 2.5x, which is 11,250 in first-attempt-equivalents. Total 25,750, and at 0.00234 each that is 60.26. Judge cost about 5.15. Total 65.41. Cost per accepted output is about 0.0090, which now exceeds Candidate A's 0.0079.

That is the decision rule worth internalizing: a candidate wins on the swap if its cost per accepted output is lower, and the cheaper token price is only a reason to test it. The break-even pass rate is computable before you run anything. Let the expensive candidate have token cost T and pass rate p, and the cheap one token cost t and pass rate q, with retry multiplier r and retry success s. Write an expression for cost per accepted output for each, set them equal, and solve for the pass rate gap at which the cheaper candidate loses. For most reasonable retry policies, a gap of more than about 10 percentage points on first-try pass rate is enough to erase a 2x to 3x token price advantage.

Cascades are the natural response. Send everything to the cheap model, score the output, escalate only failures to the stronger one. The model cascade can cut blended cost substantially when most traffic is resolved by the cheap tier (TrueFoundry). The cost per accepted output for the blend is exactly the formula above with the accepted count spanning both tiers. Measure the cascade itself, not just its parts, because the escalation threshold is a tunable that changes both terms.

Routing is not failover. Error fallback moves to another model after a failure; a quality cascade moves after a quality check. Those are different policies with different bills, and the second one is the one that changes cost per accepted output (Inworld AI).

Run the comparison as a small experiment with a decision written down in advance.

• Fix the task and the request profile. Pull a representative sample of real inputs and hold it constant across candidates. Synthetic cases fill gaps but should be labeled separately, since they often score differently than production traffic (Evidently AI).
• Freeze the pipeline. Same retry policy, same judge, same repair prompt, same tool limits. A difference in the harness will show up as a difference in the metric.
• Shadow the candidate. Run both models on the same live requests without showing the candidate's output to users, and record cost and acceptance on the same sample.
• Estimate the gap honestly. A 300 case sample with pass rates 88 and 81 percent still carries meaningful uncertainty; size the sample for the gap you care about, not for the number of requests you can afford.
• Set the rollback rule. If cost per accepted output rises more than a set percentage in the first window, or the human review pass rate falls below the bar, revert.

One decision rule that saves time: stop comparing candidates whose pass rates differ by more than roughly 15 points. The cheap one almost never recovers that gap on cost per accepted output, and the expensive one will look dominant even if its token price looks startling. Compare only candidates in the same band, then use cost per accepted output to pick among them.

Log route exposure to product analytics and join it to accepted output, satisfaction, retention or revenue, because a faster or cheaper model that lowers acceptance in the product is not a saving (Inworld AI). Cost per accepted output is a proxy for that: it captures quality at the model boundary, while product metrics capture what users do when quality changes.

• Counting only successful calls. Every failed and retried attempt belongs in the numerator, or you are measuring the bill you wish you had.
• Changing the judge or the bar mid-experiment, which makes the two candidates incomparable.
• Comparing a candidate on a request profile it does not receive. If the cheap model needs a longer prompt or more retrieved context to hit its pass rate, price that in.
• Treating an average token cost as a per-output cost. Agents often make several model calls per user turn, each with its own routing decision and cost (MLflow).
• Ignoring latency and retry latency. A cascade that escalates often can cost less per accepted output but feel slower, and the end-to-end delay includes retries (Anyscale).
• Optimizing the headline model and leaving the rest. Cost tracking works when every call carries request-time metadata that connects spend to the feature and workflow, not just the top-level model (Braintrust).
• Comparing candidates without a fixed, versioned golden dataset, so the run cannot be repeated or reviewed.

When you are ready to run the comparison across several providers, putting every candidate behind one interface keeps the harness identical and makes the cost side of the join simpler. Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, with prepaid pricing in US dollars at https://quantaintelligence.ai/inference.