Routing Ad, Page and Support Drafts to Cheapest Model

A pass mark is a number, written down before testing, that answers one question: at what score would I ship this draft with no human edit? It is not the score of the best output you have seen, and it is not the average. It is the floor.

The practical way to set it is to grade real shipped work. Pull 30 to 50 published ads, live page sections and sent support replies that a human approved without changes. Score them on a 1 to 5 rubric for your workload, and note the distribution. The pass mark is the score at or above which nothing in that set needed a rewrite. If your historical sent-minus-edits set scores 4 or 5 on 92% of samples, a bar of 4 is defensible; a bar of 3 is not, because you have shipped evidence that 3s get edited.

The rubric needs to be specific enough that two graders agree. For ad copy, separate the checks: does the headline make a claim the product can support, is the call to action unambiguous, does the length fit the placement. For page copy, add structural checks: does the section answer the question the previous section raises, does it repeat the pricing terms exactly, does it stay inside the brand's voice rules. For support drafts, add policy checks: is the cited policy current, does the reply commit to a remedy the agent is allowed to offer, does it tell the customer what happens next. Template tools such as the Langflow ad copy variation generator show how the instruction set can be made explicit in a prompt, which makes the rubric easier to apply consistently when you grade.

Two rules keep the pass mark honest. First, grade blind: strip the model name and shuffle samples so the grader does not know which tier produced which draft. Second, include your own historical best output in the set as a reference point, so the bar is anchored to what actually ships, not to an idealized draft nobody has ever written.

The test set does not need to be large. It needs to be representative, fixed and small enough to grade by hand in an afternoon. Twenty to fifty samples per workload is enough to separate a draft that routinely clears a bar of 4 from one that routinely lands at 3, and small enough that two people can score it without hiring annotators.

Build it from real inputs, not invented prompts. For ad copy, use briefs that actually ran: product, audience, benefit, tone and platform. For page copy, use the section briefs that went to writers, including the surrounding sections and the pricing or feature facts the copy has to stay consistent with. For support drafts, use real ticket text, with customer details removed, plus the policy pages the reply must respect.

Keep the set fixed across runs. If you change the samples between model comparisons, the scores are no longer comparable and the routing decision becomes guesswork. Append new samples as a separate holdout set instead, and use that holdout for the spot check later.

One input every set needs is the brief. A model can only meet a bar if the brief gives it the facts: the offer, the audience, the placement, the tone, the policy. The variation generator template above makes the brief a required, validated step before generation starts, and that structure is worth copying regardless of the tooling you use.

Test in ascending order of cost per draft, and stop at the first model that clears the workload's bar on the majority of samples. That ordering is what makes the result a decision; testing the expensive model first biases the comparison toward it.

There is a real spread to exploit. The LLM pricing comparison at CostGoat tracks hundreds of models with input prices from a fraction of a cent per million tokens to several dollars, and its value score, which divides a quality score by output cost, ranges from under 2 to over 400 across the models it ranks. The point is not to pick the model with the highest value score; it is that quality and price do not move together, so the cheapest model that clears a bar can be far cheaper than the next tier up.

Scoring needs a number per sample, not a pass or fail per batch, so you can see how close a model came. A simple weighted rubric:

Ad copy          Page copy             Support draft
claims     x3    offer match    x3     policy accuracy x4
hook       x2    structure      x2     resolution      x3
platform fit x2  claim support  x3     tone            x1
clarity    x1    reading level  x1     next step       x2
                 no price errors x3
Score each 1-5, multiply, sum, divide by total weight.

Record the share of samples at or above the bar, not just the average. A model averaging 4.2 but failing 30% of samples is not safe for a high-bar workload; the average hides the failures. Also note the failure mode: a draft that scores 3 because of one weak sentence is a different risk from one that scores 3 because it invented a feature. The second kind of failure should count as a hard fail regardless of total score.

That gives a straightforward rule. A workload route moves to a model only when the model's pass rate on the fixed set is at or above the level your historical sent-without-edit data supports, the model's hard-fail count is zero, and the cost per draft is lower than the tier above. The TrueFoundry routing overview frames routing as a ladder from static rules to cascades, and this is deliberately the bottom rung: a static rule you can read, defend and recheck, justified by measured benefit rather than adopted because it exists.

Write the routing rule as data, not as code buried in a prompt string. A small table with the workload, the pass mark, the model and the date it was validated is enough, and it makes the spot check a lookup rather than an archaeology project.

workload         pass_mark  model      validated_on  pass_rate  hard_fails
ad_copy          4.0        small      <date>        0.88       0
page_copy        4.3        mid        <date>        0.91       0
support_draft    4.5        mid        <date>        0.84       0

The spot check is the part most teams skip. On a schedule, pull a fresh batch of live drafts from the holdout set or from the last sprint's real work, score them against the same rubric and look for two things: the pass rate, and any hard fails. If the pass rate holds and no hard fails appear, keep the rule. If either slips, move that workload up one tier and leave the other two where they are. This is the three-tier cascade from the Right-Sizing the Frontier guide in its simplest form: a wrong cheap draft costs more than an unnecessary larger call, so the threshold biases toward quality until production evidence says otherwise.

Recheck when any of the inputs change, not only on a calendar. A new prompt template, a new product fact in the brief, a policy update, or a model version bump all invalidate the evidence behind the rule. When that happens, rerun the fixed set for the affected workload only and keep the other routes as they are. The update cost is an afternoon, which is why the fixed set is worth its modest size.

Suppose you produce 1,000 ad drafts, 250 page sections and 3,000 support drafts a month, and you have three tiers in mind that cost roughly $0.20, $1.00 and $4.00 per 1,000 drafts respectively. These are illustrative figures for the arithmetic, not quotes.

First run the fixed sets. Ad copy scores: small model 88% at or above a 4.0 bar, mid 94%, large 96%. Page copy: small 61%, mid 91%, large 93%. Support drafts: small 55%, mid 84%, large 91%. With the rule "cheapest model clearing the workload's pass rate with zero hard fails," ads go small, pages go mid, support goes mid.

Monthly model cost at those rates: ads 1,000 drafts at $0.20 per thousand is $0.20; pages 250 drafts at $1.00 per thousand is $0.25; support 3,000 drafts at $1.00 per thousand is $3.00. Total is $3.45. Sending all three workloads to the large tier costs $17.00. The saving is $13.55 a month in this deliberately small example, and the shape of the result is what matters: the mix sits mostly on the mid tier because two of the three workloads have high pass marks, and the ad route takes almost none of the spend.

Now suppose the spot check three weeks later finds the support route at 76% against a bar that historically shipped at 84%, with one hard fail where a draft promised a refund outside policy. The rule says move support drafts up one tier. That raises support cost from $3.00 to $12.00 at the large tier, but the hard fail is the reason, not the pass rate. In this pattern the cost of a single wrong refund commitment is likely to exceed the entire monthly delta, which is the reasoning the routing guide at Braintrust makes explicit: a cheaper route is only safe when it preserves the task's required accuracy.

The spot check that causes this correction is also where production sampling matters. The page conversion research above notes that only 13% of A/B tests reach significance, so do not wait for a conversion test to tell you a route has degraded; grade drafts directly, on the rubric, at a cadence you can keep.

The first failure is a pass mark set from aspiration rather than evidence. If no historical draft ever scored a 5, a 5 bar guarantees the route stays on the largest model forever. Anchor to sent-without-edit work.

The second is a test set that drifts. Mixing new samples into the fixed set between runs makes a 3% improvement indistinguishable from a change in the samples. Keep the fixed set and the holdout set separate.

The third is grading with the model name visible. Grader expectations move when they know a small model wrote a draft, and the result is a bar that quietly favors the expensive tier. Blind grading is the cheapest control you have.

The fourth is treating routing as failover. A route that fails its spot check should move up a tier and stay there until the fixed set says otherwise, not fall back to the big model for one bad sample and return the next day. The distinction is spelled out in the TrueFoundry write-up: routing is a policy, failover is availability, and mixing the two leaves you unable to explain either.

The fifth is ignoring the brief. If the ad brief omits the proof for a claim, no model can clear a claim-accuracy check, and the route looks worse than the model is. Fix the brief before you blame the tier.

The rule above is model-agnostic, which means the practical bottleneck is access to a range of tiers without wiring a different client for each provider. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key with prepaid pricing in US dollars, so trying a small, mid and large tier on the same fixed set is a config change rather than an integration project: Quanta Inference.

If the grading itself is the slow part, that is a good place to bring in help. A working session with an AI expert can turn your rubric, fixed set and spot check schedule into a repeatable process your team owns rather than a one-off experiment: Quanta Experts.