When a Human and a Second Model Disagree on Copy

The rule that removes most of the friction is simple. For every rubric item, decide in advance which judge wins if the two disagree. Write that winner next to the item. When a disagreement lands, nobody debates who is better in general; the item itself names the winner.

The logic for assignment is what the item measures:

• Items about facts, names, numbers and dates go to the judge that can be checked against a source.
• Items about policy, brand voice, legal exposure and tone go to the person.
• Items about mechanics (length, formatting, spam triggers, link validity) go to the model, because it applies a fixed test every time.
• Items about whether the copy earns a reply go to the person, unless you have response data, in which case the data wins and neither judge is consulted.

This is a pragmatic reading of what the data says about where each judge is strong. Personalized subject lines are reported to lift open rates by 26 to 50 percent over generic ones, and subject lines under seven words outperform longer ones in B2B outreach (Cleanlist's subject line roundup). That is a mechanical claim a model can check without taste. Whether a specific opener fits a specific prospect is a judgment call.

A worked example makes the assignment concrete. Suppose a review covers one cold email and the rubric has six items. The human scores the opener 3 out of 5 and the second model scores it 5 out of 5. The pre-assigned rule says: opener relevance to a checked fact goes to the person. The final call is 3, and the model's 5 is recorded but overridden. The reviewer spends no time arguing because the rule already decided.

The table below is the artifact that makes the whole system work. Fill in the winner column once, for the rubric you actually use, and pin it where reviewers can see it. The "why" column is the teaching layer: it explains the choice to a new reviewer so they apply the rule confidently.

Two features are load-bearing. First, every row has exactly one winner, so no row ends in a debate. Second, the escalation column is narrow: it names the specific condition that pushes a disagreement past the rule. A rule that escalates everything is the same as no rule.

A note on why "model wins" is safe for mechanical items. When you separate what good cold email mechanics look like from whether a particular opener lands, the mechanical part is the part that repeats. Practitioners writing personalized first lines keep them to one or two short opening sentences, roughly 10 to 30 percent of the email, which is a countable property (Cleverviral's workflow). Countable properties belong to the judge that counts.

Escalation is the expensive path, so reserve it. A disagreement should reach a third person only when one of these is true:

1. The rubric item has no pre-assigned winner. That is a gap in the table, and the escalation should produce a new row, not just a verdict on one email.
2. The item is high-stakes and the two scores straddle the threshold that decides whether the copy ships. A 2 versus 3 on a tone item for a major account is a real escalation; a 4 versus 5 is not.
3. The person and the model disagree on a fact, and the person cannot resolve it from a source. That is a data problem that a reviewer should not paper over.
4. The item touches legal, regulatory or contractual language. This one always escalates, every time, because the cost of a wrong call is not recoverable by editing later.

The reason to keep the list short is human attention. Oversight of an AI system has real costs in time and cognitive load, and the design of the loop determines whether the human oversight is meaningful or ceremonial (the human-in-the-loop review). If everything escalates, reviewers skim, and the escalation stops catching anything.

When you do escalate, treat the disagreement as data. The literature on subjective annotation argues that disagreement is not merely noise and that modeling who disagreed and why can improve the system downstream (Prolific on reading disagreement as data). Practically, that means the escalation record should say which judge was right and why, so the rule table can absorb the lesson.

The log is where the time savings compound. Every resolved disagreement should produce one row with four fields, plus a little context.

• The item: which rubric line, and the exact text or excerpt it scored.
• Both judgments: the human score and the second model's score, with the judge identifier.
• The rule used: which row of the table applied, or "escalated: no rule" if none did.
• The final call: the score that shipped, and who or what made it.

Add a fifth field for the reason in one line, and a sixth for a link to the source if a fact was involved. Keep the log in the same system as the copy so a reviewer can search it.

A minimal record looks like this:

{
  "item": "opener_tied_to_checked_fact",
  "human": {"judge": "reviewer_04", "score": 3},
  "model": {"judge": "judge_b", "score": 5},
  "rule_applied": "fact_items_human_wins",
  "final_call": 3,
  "reason": "model cited a funding event not present on the company site",
  "source": "https://example.com/company/news",
  "review_id": "rev_2024_118"
}

The point of the structure is retrieval. When the same kind of disagreement shows up again, a reviewer searches the item name, finds the prior row, reads the reason and applies the same outcome. That is the whole mechanism by which the second review is faster than the first. It also gives you the raw material for fixing the rubric: if one item keeps escalating, the rule table is telling you the item is under-specified.

Two disciplines keep the log honest. First, record the model's score even when it loses, because the pattern of model overrides is itself information about where the model is overconfident. Second, never edit a past row. Append a correction instead, so the audit trail stays intact.

A tie-break rule only helps if the two judgments are genuinely independent. If the second model is the same model that drafted the copy, you are not getting a second opinion; you are getting a rewrite. Use a different model for judging than for drafting, and ask it to score against the same rubric the person uses, not against its own taste.

The same independence applies to the person. If the person wrote the copy, their score is not an independent judgment. Route the human review to someone who did not write the draft, or accept that the human score carries less weight on items where the writer is emotionally invested.

There is a practical wrinkle worth naming. Model judges on subjective axes are unstable and vary by judge model, which is exactly why the per-item assignment matters: you are not trusting the model's taste, you are trusting its ability to run a fixed check (Prolific on judge instability). Keep the judge model pinned to a specific version and record that version in the log. If the judge changes, the rule table may need a fresh look.

When you want to test whether a judge change actually improved agreement rather than just shifted it, measure it. Comparing two labeling schemes on the same items with a chance-corrected statistic tells you whether the apparent improvement survives chance. The guidance to report confidence intervals and describe the rater design applies directly to model judges (the IAA survey). A single agreement number without uncertainty overstates precision.

The disagreement problem gets bigger as you add judges. A pipeline with a drafting model, a scoring model and a human reviewer is normal for serious copy work, and the tie-break table scales to it: you just add a winner per item across more than two judges. The cost is that you now have to route prompts to several different models and track which one produced which score.

That routing is the friction point. If your judges live behind different APIs with different request formats, the review pipeline inherits that complexity. Quanta Inference puts text, image and embedding models behind one OpenAI-compatible API, so a drafting model and a separate judging model can be called from the same code path with one key, which keeps the judge independent without a second billing relationship.

The other leverage point is the review itself. The work of applying a rubric and logging the decision is repetitive, and it is exactly the kind of task that benefits from a specialist who reads your rubric and context and returns a finished, structured result. That is the pitch for Quanta Experts: an expert that can take your rubric plus a batch of copy and hand back the scored items in the same log format described above, so the human reviewer spends time on the escalations rather than the routine pass.