Two Raters, One Draft: Writing a Voice Rubric They Score Alike

Keep the rubric to three rows. More rows mean more places to disagree and longer scoring sessions; fatigue is a documented source of drift in essay scoring, where a scorer's severity shifts as a session lengthens inter-rater reliability in essay scoring. Three rows on a 1 to 5 scale take a rater about 90 seconds per draft once calibrated.

The separation matters because the rows fail independently. A draft can nail the voice and contain no specifics; a draft can be specific and end with three competing calls to action. If you collapse those into one score, a strong voice masks a broken ask, which is the halo effect: surface quality on one dimension inflating the score on another inter-rater reliability in essay scoring.

Write the row definition as a question the rater can answer from the page. "Does this draft sound like the brand?" is not answerable. "Does this draft use second person address, sentences averaging under 20 words, and zero exclamation marks?" is.

Anchors are the only part of the rubric that determines whether two raters agree. Write anchors at 1, 3 and 5. Do not write 2 and 4; they are interpolations, and leaving them blank forces the rater to commit. A rater who wants a 2 must decide whether the draft is closer to the 1 description or the 3 description, which is exactly the judgment you want.

Here is the pattern, using voice fit:

VOICE FIT
1 = Third person throughout, or first person plural where the brand uses "you".
    Three or more exclamation marks, or two or more superlatives ("best", "revolutionary").
    No direct address to the reader in the first two sentences.
    Ends on a hedged sentence ("it depends", "results may vary").

3 = Second person address in most sentences, with one or two slips into
    first person plural. One exclamation mark or one superlative.
    Sentences average 20 to 30 words. Opens with a statement about the reader.

5 = Second person address in every sentence, no exclamation marks and no
    superlatives, sentences averaging under 20 words, at least one sentence
    under 12 words. Opens with a statement about the reader and closes on one.

Every clause is checkable by a second person with a word counter. That is the test: if a rater cannot point at the words that earned the score, the anchor is not done. The evidence on rubrics is consistent here: reliability improves when criteria are analytic, topic-specific and paired with exemplars or rater training, and a rubric alone does not guarantee valid judgment rubric design and scoring review.

Build a small exemplar bank alongside the anchors: two drafts per row scored 1, 3 and 5, agreed by both raters, with the deciding sentences highlighted. Raters should read the exemplars before each scoring session, not once at the start. The same essay-scoring literature notes that norming decays, and a session held weeks before scoring has limited effect on drafts read at the end of a long day inter-rater reliability in essay scoring.

One discipline worth borrowing from legal prompt work: instructions and constraints belong in the prompt, not in the rater's head, and the output should be verified against the stated criteria rather than against a general impression of quality Clio's guide to legal prompt engineering. Treat the rubric the same way. If a rule matters, it goes in the anchor text.

Decide this before anyone scores anything:

• If the two raters' scores on a row differ by 0 or 1 point, the row score is the mean of the two, rounded down.
• If they differ by 2 points or more, a third rater scores that row only, without seeing the first two scores.
• The final row score is the median of the three.

The third rater sees only the row and the draft, not the other scores, so the decision stays independent. Median, not mean, because with three values the median is always one of the three actual scores, which makes your regression results readable in the rubric's own units.

Double scoring with adjudication when raters diverge beyond a threshold is the standard approach in large-scale writing assessment; the tradeoff is that it roughly triples the scoring workload inter-rater reliability in essay scoring. Your calibration round exists to drive the number of disputes down so this rule fires rarely. If it fires on more than about one row in ten, you have a rubric problem, not a scoring problem.

Record the dispute rate per row. A row with a 20 percent dispute rate is not ready to gate a release, regardless of what the averages say.

The calibration set is a fixed sample of 10 to 15 drafts, deliberately chosen to span the range: two that clearly score 1 on voice fit, two that clearly score 5, and the rest ambiguous. Ambiguous drafts matter most because that is where raters diverge. Pull them from real outputs, not from drafts you wrote to make a point.

Procedure:

1. Both raters score all 10 to 15 drafts, all three rows, independently, in one sitting, with no discussion and no shared spreadsheet.
2. Neither rater sees the other's scores until both are submitted.
3. Compute exact agreement and within-one-point agreement per row.
4. List every draft and row where the gap is 2 points or more and read the anchors against the text together.
5. Revise the anchor text where the gap traces to a missing clause, then rescore the revised rows on the same sample.

With 12 drafts, each row gives you 12 paired scores. Keep the arithmetic simple and show it in the room:

Voice fit, 12 drafts
Exact matches: 7
Gaps of 1:     4
Gaps of 2+:    1
Exact agreement: 7/12 = 58%
Within one:     11/12 = 92%
Dispute rate:   1/12 = 8%

Compute this in a script or a sheet, not by eye, and keep the sheet versioned with the rubric. Anyone rerunning the set later needs to know which anchor text produced which numbers.

There is no universal pass mark, but the interpretation of kappa coefficients gives you a defensible scale: below 0.40 is poor, 0.40 to 0.60 moderate, 0.60 to 0.80 substantial, and above 0.80 nearly perfect, with most high-stakes assessment aiming above 0.70 inter-rater reliability in essay scoring. Clinical rubric work reached similar conclusions: a documentation rubric can show acceptable reliability, but only after the criteria are tightened and raters are trained on them inter-rater reliability of a clinical documentation rubric. Rubric development studies reach the same conclusion about validity needing explicit estimation rather than assumption analytical rubric validity and reliability study.

For a voice regression set, exact agreement is more useful than kappa as a working number, because raters will cluster on 3 and kappa punishes clustering heavily. Use this decision table:

The within-one-point figure is your safety net, not your gate. High within-one agreement with low exact agreement means the raters agree on the band but not the rung, which is usually a missing clause between the 3 and 5 anchors rather than a real disagreement about quality. Fix the clause; do not average the problem away.

Most 2-point gaps fall into four patterns. Name the pattern before editing anything.

• Boundary gaps. One rater scores 3, the other 5, on the same draft. Cause: the anchors describe endpoints but not the midpoint transition. Fix: add a clause to the 3 anchor that says what keeps a draft out of 5.
• Feature-count gaps. One rater counts a feature the other did not see. Cause: the anchor says "concrete examples" without a number. Fix: state the count.
• Definition gaps. One rater treats a product name as a specific, the other does not. Cause: no glossary. Fix: add a two-line glossary under the row.
• Order effects. Raters score a strong draft and then score the next one low. Cause: no independent session ordering. Fix: randomize draft order per rater and score in batches of five with a break.

Rater severity is the last thing to fix, not the first. A rater who is systematically one point harsher on specificity is easy to correct with a stated offset after calibration and easy to detect from the distribution of their scores. A rubric with a vague anchor cannot be corrected at all.

Stop calibrating when a fresh sample of five drafts produces no 2-point gaps and exact agreement on the sample is at or above your threshold. Then freeze the rubric text with a date and a version label and start scoring the real regression set. Revisit whenever you change the prompt template, because a prompt change moves the outputs and the anchors may no longer describe the new distribution.

If your regression set ends up needing many prompt variants and models compared side by side, running them through a single endpoint keeps the scoring pipeline in one place; Quanta Inference gives one API key across text, image, video, audio and embedding models, OpenAI-compatible and prepaid, which removes per-provider plumbing from the evaluation harness (https://quantaintelligence.ai/inference). For teams that want the regression set built and operated rather than hand-rolled, a Quanta expert can take the rubric, calibration data and scoring sheet and run it as a standing process (https://quantaintelligence.ai/agents).