A Scorecard for Choosing Which Model to Call

The scorecard reuses the five criteria from the earlier matrix. What changes here is the scoring rubric. Without anchored descriptors, two reviewers will score the same model three points apart and the total means nothing. Most practitioners who write about weighted scoring recommend a 1 to 5 scale with a written description for each level to keep scoring consistent according to the AI use-case prioritization framework write-up from Umbrex.

Three notes on filling this in. Quality scores must come from your own evaluation set, not from a public leaderboard, because the leaderboard tasks are not your tasks. Latency should be measured the way the vendor's own benchmarking guidance defines it: time to first token and end-to-end latency are separate numbers, and both grow with prompt length and concurrency per NVIDIA's explanation of inference benchmarking concepts. Cost per call should be computed from your median input and output token counts, not from a price list per token, because prompt length drives the bill.

Data terms deserve a warning. Score this criterion on the contract you have, not the marketing page. If a candidate is disqualified on data terms for a given workload, that is a gate, not a low score: remove it from the row rather than letting a strong quality score drag it back through the average. Gating conditions that screen options out before scoring are a standard part of the framework described by Umbrex.

Weights are where the workload enters the card. Before scoring anything, write one sentence describing how the output is used, and let that sentence set the weights. A useful discipline from project prioritization frameworks is to rank objectives first and let the ranking produce the weights, rather than assigning percentages by feel as Fountain City's scoring framework argues.

Read the table as a set of arguments, not a prescription. The chat assistant gives latency a third of the total because a slow answer is a broken product no matter how good the text is. The regulated extraction workload gives data terms 40 percent and latency almost nothing, because a batch that finishes in an hour is fine and a retention clause that conflicts with the contract is not. The agent workload weights fallback heavily because a multi-step agent that dies halfway through a task leaves work in an inconsistent state, and recovery matters more than a marginal quality gain.

Two failure modes to watch. Flat weights (20 percent each) are almost always wrong and hide the decision; they mean you have not decided what the workload needs. And weights that drift after scoring are just a way of reversing a result you dislike. Freeze the weights before the first score is entered, and log any change with a reason and a date.

Suppose a support team is choosing a model for an assistant that drafts replies to customer emails. Median input is 1,800 tokens and median output is 250 tokens. Volume is 40,000 drafts a month. The acceptance bar, set in advance, is that a draft must be rated acceptable by a reviewer at least 70 percent of the time without edits.

The team has three candidates already shortlisted and screened on data terms. Weights: quality 30, latency 30, cost 10, data terms 15, fallback 15.

The arithmetic for Model B is (4 × 0.30) + (5 × 0.30) + (5 × 0.10) + (4 × 0.15) + (3 × 0.15) = 1.20 + 1.50 + 0.50 + 0.60 + 0.45 = 4.25. Model A wins on quality and fallback but loses badly on latency and cost. Model C is competitive on cost and terms but sits below the acceptance bar and has the weakest failover.

Now change one weight and watch the answer move. If the workload is a nightly bulk rewrite of help-center articles rather than a live assistant, latency drops to 5 percent and cost rises to 40 percent:

Model B still wins, but the margin over Model C narrows to 0.30, which is inside the noise of a five-point rubric. That is the moment to look at the underlying numbers rather than the totals: if Model C can be lifted over the acceptance bar with a post-check that costs less than the price gap, it becomes the right call. Cost per call is not only the token price; the run book for two candidates is the same, the difference is in retries, post-processing and review load. A recent study of small business AI spending found that newer adopters start at lower spending levels driven by accessible entry-level services per JPMorgan Chase Institute research, which is a reminder that the cheapest workable option is often the right starting point when volume is uncertain.

A scorecard decays fast if the cells contain numbers without provenance. Attach one evidence line to every cell that is not a 3, and keep the evidence in the same repository as the routing code.

For quality, the evidence is an evaluation run: dataset name, number of cases, reviewer or grader, and the date. For latency, it is a load test at your real concurrency, reporting time to first token and end-to-end latency separately as defined in NVIDIA's benchmarking metrics. For cost, it is a monthly estimate computed from median token counts times unit price, with the price source and the date it was read. For data terms, it is the clause reference and the reviewer's name. For fallback, it is the date of the last failover drill and how long the switch took.

Keep the card in version control next to a small routing config so that the decision and its implementation cannot drift apart:

# model_scorecard.yaml
workload: support_reply_drafting
weights:  { quality: 0.30, latency: 0.30, cost: 0.10, data_terms: 0.15, fallback: 0.15 }
weights_frozen_on: <date set before scoring>
candidates:
  model_b:
    scores: { quality: 4, latency: 5, cost: 5, data_terms: 4, fallback: 3 }
    total: 4.25
    evidence:
      quality: "eval set support-tickets-v3, 600 cases, 2 reviewers, pass 74%"
      latency: "load test at 20 concurrent, p95 first token under target"
      cost: "median 1800 in / 250 out tokens, price read from provider page"
      data_terms: "DPA clause 4.2 reviewed, no training on inputs"
      fallback: "secondary path defined, drill pending"
selected: model_b

Two governance habits keep the card honest. First, record the weights and the scores before the final ranking is shown to the person who requested the project; a card built after the answer is known is a rationalization. Second, schedule a review trigger on price changes, contract renewals and major capability releases. Prioritization frameworks are explicit that scores and weights get revisited between planning cycles as Savio's weighted scoring guide notes, and model catalogs change faster than product roadmaps do.

It is also worth scoring at the routing level, not the model level, once you have more than one workload. A candidate that ranks second on every individual workload can still be the best default if it keeps the number of integrations and contracts small, because operational simplicity is a real cost that the five criteria only partly capture.

A scorecard ranks candidates. It does not tell you whether the workload should exist. That question belongs to the project selection stage, where value and feasibility are assessed before any model is compared, and where projects that lack a named owner for the decision should be deferred rather than scored per the gating conditions in Umbrex's framework. If the business case is thin, a perfect model comparison just gets you to the wrong place faster.

Three practical limits to keep in mind. The first is that quality scores are only as good as the eval set, and a small or unrepresentative set will produce confident nonsense. The second is that weights are judgments, and the tool's critics are right that weight assignment is subjective as Lucid's discussion of the method acknowledges; the fix is to have the people who own the workload set the weights, in writing, before scoring. The third is that a five-point rubric compresses real differences. A total gap under about 0.3 on these weights is not a decision, it is a prompt to look at the raw numbers.

For teams running several workloads across several model families, the operational overhead of the card itself starts to matter. That is the point where consolidating access helps: Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key with prepaid US dollar pricing, which keeps the cost column in the card comparable across candidates rather than split across five billing relationships.