When to Route a Copy Review to a Cheaper Model as Second Judge

The policy is one grid. Read down the second-judge column: that is what runs after the first judge has already scored the item.

The rule in one sentence: the second judge is cheap only when the item can be fixed after publication without a refund, a correction notice, or a regulator's letter.

The middle of the table is where teams get burned. Email and hero copy feel creative, so they get routed like social captions, but they reach people you cannot retract from. Use "can this be silently edited later?" as the sorting question and the table fills itself in.

Agreement is measured, not assumed. Run a calibration sample per content type before routing anything to the cheap judge, and rerun it when the rubric, the brand voice guide, or either model changes.

1. Pull 100 to 200 items from the last few weeks for one content type only. Include at least 20 items a human editor previously rejected, because a sample of clean approvals flatters any judge.
2. Have the strong judge score every item against the rubric and record pass or fail plus the reason.
3. Have the cheap judge score the same items with the same rubric, same reference facts, same policy text. Do not shorten the cheap judge's prompt to save tokens; prompt trimming is a common cause of fake disagreement.
4. Compute agreement two ways: raw percentage and a chance-corrected coefficient.

from sklearn.metrics import cohen_kappa_score

strong = [1, 1, 0, 1, 0, 1, 1, 0, 1, 1]   # 1 = pass
cheap  = [1, 0, 0, 1, 0, 1, 1, 1, 1, 1]

raw = sum(a == b for a, b in zip(strong, cheap)) / len(strong)
kappa = cohen_kappa_score(strong, cheap)
print(f"raw agreement {raw:.2f}  kappa {kappa:.2f}")

Report both numbers. Raw percentage agreement is intuitive but it does not correct for agreement that would happen by chance, and it can overstate reliability when categories are imbalanced the inter-annotator agreement metric paper. Copy review is usually imbalanced: most items pass, so two judges that approve almost everything will look like they agree.

Chance-corrected coefficients bring their own problem. Cohen's kappa and Fleiss' kappa assume each item has one correct label and treat every departure from consensus as error Prolific on subjective tasks. Copy review is often genuinely subjective, since whether a headline is too clever admits more than one defensible answer. Use the coefficient as a floor test, not as truth.

A workable threshold set for a copy pipeline:

• Raw agreement at or above 90 percent on one content type is the gate for cheap routing.
• A chance-corrected coefficient well below the usual interpretive range on the same sample points at an underspecified rubric, not a bad cheap judge. Fix the rubric before changing the routing.
• Count the two disagreement directions separately. Cheap passes while strong fails is the dangerous direction; cheap fails while strong passes costs rework, not a brand incident.

The disagreement pattern matters more than the headline number, which is why agreement reporting guidance recommends analyzing disagreement patterns and reporting confidence intervals rather than a bare point estimate the IAA metric paper. With 100 items, a 90 percent figure carries a margin of error of several points, so treat 90 percent as borderline and collect more items.

Take a content operation shipping 10,000 judged copy items a month: 6,000 social captions, 2,000 blog subheads, 1,200 internal notes, 500 landing page and ad items, and 300 email and regulated items.

Assume, for illustration only, that a judged item costs 20 units with a strong judge and 2 units with a cheap judge. These are arithmetic for a model of the policy, not observed prices; substitute your own invoice figures.

Three things fall out of the table.

The audit line is not free. A 5 percent audit of roughly 9,200 cheap-judge approvals costs about as much as all 500 landing page items judged at full strength. That is the price of the control, and it belongs in the budget before anyone celebrates the savings.

The landing page row shows that cheap judge plus audit can cost more than strong-only when volume is low. Routing saves money only if the volume through the cheap path is large enough to amortize the audit. A useful break-even test: route only when cheap cost plus audit cost is below strong cost for that content type at that monthly volume.

Savings concentrate in a few high-volume, low-stakes types. That matches where cascaded judging pays: the Cascaded Selective Evaluation work uses cheaper models as initial judges and escalates to stronger models only when needed, and reports a provable human-agreement guarantee with high coverage while using substantially cheaper models the ICLR paper. The gain comes from coverage, not from shaving every item.

Set the cap explicitly. A common shape allows the cheap path no more than 40 percent of the monthly review budget, with the rest reserved for strong-judge work, audits, and escalation headroom. If cheap-path spend breaches the cap, the policy has drifted: either volume grew or the audit rate crept up.

Routing is safe only if escalation is precise. A second judge that can pass an item and end the process is a single point of failure on the cheap side.

Build escalation into the rubric rather than prose:

• Hard triggers, always escalate: any number, price, percentage, statistic, award, certification, guarantee, comparative claim, competitor name, named individual, health or financial claim, or claim about a child.
• Confidence triggers: escalate when the cheap judge returns a borderline score or an unstructured answer. A judge that cannot express uncertainty cannot be trusted on borderline items.
• Source-missing triggers: if the reference facts for a claim are absent from the judge's context, escalate rather than pass. An unsupported claim is the most common failure a judge catches when it is given source material the Arize guide.

It helps to know what the research says about judge reliability on subjective axes. Judges are unstable on the very subjective axes that matter most and favor outputs resembling their own Prolific's summary of the evidence. That is an argument for keeping the strong judge on tone-sensitive, high-visibility copy even when mechanical checks agree.

Adjacent automation makes the same point from a different direction. A cold email pipeline that scrapes a company's pages and summarizes them before writing a first line works because the model is handed checked, specific source facts rather than a vague instruction a walkthrough of that workflow. Copy judges behave the same way: give both judges the same verified facts and the same rubric and agreement becomes meaningful. Starve the cheap judge of context and you measure prompt poverty, not model weakness.

The prompt side matters equally. Generic prompting produces generic output because the model has no real information to work against a workflow guide for personalized first lines. Judge prompts face the identical constraint.

The audit is the control that keeps the policy honest. Without it, the cheap path quietly becomes the whole pipeline.

Run it weekly on a random 5 percent of items the cheap judge approved. The strong model re-judges them blind, without seeing the cheap verdict. Track four numbers:

1. False-approval rate: the share of audited approvals the strong judge rejects. This is the headline quality metric for the cheap path.
2. Category concentration: the escalation triggers most often missed. If most false approvals are pricing claims, the cheap judge is systematically weak on numbers.
3. Segment drift: false-approval rate per content type. One bad week on one segment should pull that segment back to strong review, not the whole pipeline.
4. Audit coverage: audited items divided by the number you intended to audit. A 5 percent target is worthless if the sample is the last 5 percent of each batch, which is not random.

Set response rules in advance:

• False-approval rate above 2 percent on a content type: pause cheap routing for that type and rerun calibration.
• Any false approval on an item that carried a hard trigger: add the trigger to the cheap judge's escalation list and audit that trigger at 100 percent for the next two weeks.
• Any false approval involving regulated content: pull the whole category back to strong-only and review the routing table itself.

This is a lightweight version of human-in-the-loop discipline, where oversight is placed at the points in a pipeline where error costs are highest a systematic review of human-in-the-loop AI. Copy review is low stakes compared with clinical or lending decisions, but the design pattern is the same: automate the cheap path, put oversight where a mistake is expensive, and measure whether the oversight works.

One measurement trap: open rates and clicks are not quality metrics for a judge. If the second judge exists to catch false claims, measure false-claim escapes, not engagement. If you do weigh engagement, treat subject line research as a source of hypotheses rather than a target. Personalized subject lines correlate with higher open rates in large send studies GigRadar's summary of subject line data, but the effect is correlational and open metrics are degraded by privacy features that mask opens Klaviyo's guidance on Apple Mail Privacy Protection. Judging copy against a noisy engagement metric trains the cheap judge to approve clickbait.

Routing on vibes. "It's just a social post" is not a policy. Write the content type list, the trigger list, and the agreement floor where the pipeline can read them.

Global agreement instead of per-type. A judge pair can agree at 95 percent on social captions and 78 percent on product claims in the same week. Routing on the average puts both on the cheap path. Measure and route per content type.

Shortening the cheap judge's prompt after calibration. Shorter prompts save tokens and destroy the calibration you measured. If you change the cheap judge's context, recalibrate.

Treating a single coefficient as truth. A low coefficient can mean the rubric is vague, not that the judge is unreliable the metric selection paper. Read the disagreement list by hand before blaming the model.

Using a second judge from the same family as the writer. Judges can favor outputs that resemble their own Prolific's survey of the evidence. A cheap judge fine-tuned on the same tonal data as the writer model is a friendly reviewer, not an independent one.

Never raising the audit rate after an incident. The audit rate should respond to measured risk: raise it after any miss that reached a customer, lower it only after two consecutive clean weeks.

Where the pipeline needs several model families behind one key, Quanta Inference serves text, image, and embedding models through a single OpenAI-compatible API with prepaid US dollar pricing, so the strong judge, the cheap judge, and the writer are billable from one place. Teams that would rather hand the whole routing and audit loop to someone else can start with a Quanta expert at https://quantaintelligence.ai/agents.