How to Train a Second Reader on Your Go/No-Go Rule

Ambiguity about the output format wastes the calibration pass. Before you start, fix the output to three things per case:

1. The verdict: GO, NO-GO, or BORDERLINE.
2. The controlling criterion: the one criterion that decided the call, named by its number.
3. A one-line reason in the reader's own words.

That is all. Reasons should be short, because long narratives hide disagreement. The controlling-criterion field is the most useful part: when two readers disagree on the verdict but agree on the controlling criterion, the problem is usually in how the threshold is worded. When they disagree on the controlling criterion itself, the problem is in the decision order or in how evidence is read. Those are different fixes, and separating them saves you from rewriting the wrong paragraph.

Borrow the discipline from QA calibration workflows: multiple evaluators score the same case independently against a benchmark, results stay hidden until all have submitted, and then a side-by-side comparison shows exactly which criterion diverged and by how much (Intryc). The point of "hidden until submitted" is that nobody anchors on someone else's score before writing their own.

Also fix the evidence rule. State whether the reader may look outside the case file (searching the vendor's site, asking a colleague, checking past decisions). Pick one: either the case file is the whole world, or outside lookups are allowed and must be listed as "extra evidence" on the score sheet. Mixing the two creates disagreements that look like judgment differences but are really sample differences.

None of this requires software. A shared spreadsheet with one row per case, columns for verdict, controlling criterion, and reason, and scores kept in separate tabs until the reveal is enough for a team of two.

Three examples is the minimum that covers the space. You need:

• A clear pass. Every mandatory criterion passes, no hard stop, high score. Purpose: show the reader what "GO" looks like when nothing needs weighing, and reinforce that a pass does not need a narrative.
• A clear fail. A single hard stop fires, or two mandatory criteria fail. Purpose: show that hard stops end the analysis, and that a fail does not require a long write-up either.
• A borderline case. No hard stop, all mandatory criteria pass, but the score sits at or near the threshold, or one mandatory criterion is genuinely contested. Purpose: this is where the reader will guess, so this is where your narration has to be most detailed.

Choose the borderline case from real history. Pull a decision where you personally hesitated, and choose one where the hesitation had a cause you can articulate (for example, one criterion was met on a technicality). Do not manufacture a borderline case by splitting a difference; synthetic cases teach the wrong lesson because they lack the messy details that caused your hesitation.

Write each example as a case file, not as a summary. Include the same attachments and fields a real case would have, because what the reader learns is partly how to find evidence, not just how to weigh it.

Then add a fourth example: a case that looks like a borderline but is actually a hard stop. This pairing is the highest-value thing in the whole hour. Readers who have only seen clean examples tend to over-weigh everything that looks close, and the look-alike teaches them to check hard stops first instead of last.

Telling is not the same as showing. Walk through each of the four examples in the same order every time:

1. Read the case facts. Do not interpret yet.
2. Check every hard stop, in order. Say the name of each one and the result. "Conflict of interest: no. Data missing: no. Legal blocker: no."
3. Score each criterion out loud. Say the number and the evidence you used. "Criterion three, signed scope document: absent, fail. Criterion four, delivery capacity: passes, we have the team."
4. Name the controlling criterion. "This one is decided by the missing scope document, criterion three."
5. State the verdict and one line of reasoning.

At each step, stop and point to the exact sentence in the case file that carried the decision. The reasoning behind a call is rarely obvious from the facts alone; aviation decision training makes the same point in reverse, warning that the hardest part is not recognizing a problem but acting on it, because acting means changing plans (AOPA). Your reader has the same temptation in mild form: recognizing a failing criterion and then quietly ignoring it because the overall case looks attractive.

While you talk, write down every place you had to add context that is not in the rule document. Each of those is a gap in the rule, and each one becomes a candidate rewrite after calibration. Expect to find a few. A rule document that has never been taught to a second person almost always has three or four of these hidden dependencies.

Let the reader interrupt with questions during the walkthrough, and write the questions down verbatim. A question like "what if the missing document arrives next week" is gold: it is a signal that the rule does not say whether a criterion is evaluated at submission time or at decision time. Answer it in the moment, then fix the document.

Now hand over ten to twenty fresh cases. Ten is enough to find a structural disagreement; twenty gives you a more stable estimate of agreement and is worth the extra twenty minutes if the first ten show close alignment. Choose cases that look like the real distribution, not three passes and fifteen fails, unless your real traffic is shaped that way.

Keep your own scores hidden until the reader submits. This is not a formality. Once a reader sees that the lead scored a case NO-GO, the next contested case gets scored toward no-go. The blind rule is what makes the agreement number mean something.

When both sets are in, compute two numbers:

• Percent agreement. The number of cases where the verdicts match, divided by the total.
• Cohen's kappa. Agreement corrected for what two readers would hit by chance given how often each said GO and NO-GO. The formula is straightforward: subtract chance agreement from observed agreement, and divide by one minus chance agreement (Numiqo).

Here is a worked example. Twenty cases, verdicts GO or NO-GO (borderline verdicts get resolved to one or the other before comparison, with the borderline flag kept in a separate column). The two readers agree on 18 of 20, so observed agreement is 0.90. Suppose the lead said GO on 10 cases and NO-GO on 10. The second reader said GO on 9 and NO-GO on 11. Chance agreement is the sum of the products of each reader's proportions: (0.50 × 0.45) + (0.50 × 0.55) = 0.225 + 0.275 = 0.50. Kappa is (0.90 − 0.50) / (1 − 0.50) = 0.80. That is strong agreement.

Now the same 90 percent, with a skewed reader. The lead said GO on 18 of 20. The reader said GO on 17 of 20. Chance agreement is (0.90 × 0.85) + (0.10 × 0.15) = 0.765 + 0.015 = 0.78. Kappa is (0.90 − 0.78) / (1 − 0.78) = 0.12. Same 90 percent agreement, but almost all of it is explained by both readers saying GO nearly all the time. This is the kappa paradox, well documented when nearly everyone gets the same rating (The Analysis Factor). The practical consequence for you: on a go/no-go rule, percent agreement alone will flatter you. Track kappa too, and keep the case mix from becoming all passes or all fails.

On interpreting kappa, use the common benchmark that about 0.70 indicates good reliability, while noting that the threshold depends on the stakes. A useful rule of thumb is that a score used to decide whether to fund something consequential should sit well above the cutoff that a low-stakes screening score can tolerate (McHugh, Biochemia Medica).

Hold the review meeting to a strict agenda. Do not re-discuss the cases where you agreed, even the interesting ones. Two things happen otherwise: the meeting expands to fill the hour, and the reader learns nothing, because the agreements already worked.

For each disagreement, ask three questions in this order:

1. Did you and I read different facts? If so, the issue is the evidence rule or the case file, not the rule itself.
2. Did you and I check the criteria in a different order? If so, the decision tree needs explicit ordering.
3. Did you and I interpret the same criterion differently? If so, the criterion wording needs a concrete anchor: a number, a threshold, or a worked sub-example.

Then change one thing. Usually the one thing is a single criterion's wording, or the placement of a hard stop. Changing five things at once makes the next calibration round uninterpretable, because you will not know which change moved the agreement.

There is a real risk in this step, and it is worth naming. When a disagreement is open, the temptation is to adjust the instruction toward whichever reader complained loudest, which makes the rule fit one person rather than the work. The discipline Intryc describes for QA scorecards applies directly: calibrate the humans before you touch the instruction, because editing a rule while a disagreement is live drags it toward whoever wrote the edit (Intryc).

The same caution appears in performance review calibration, where the goal is stated as making sure all managers apply the same standards so that one manager does not rate consistently higher or lower than another for comparable work (Leapsome). Sub in "reader" for "manager" and the principle carries over exactly.

Rerun the calibration with a fresh batch after each rewrite. Stop when you agree on at least 90 percent of cases and kappa is comfortably above the chance-corrected floor you set. Then move from calibration mode to spot checks: every tenth case, or every case where the reader marks BORDERLINE, gets a silent second score.

The one-hour session gets you a second reader who scores blind. Keeping that alignment takes about fifteen minutes a month:

• Pull five recent cases where the post-hoc outcome is now known, and compare both readers against the outcome. This separates agreement from accuracy. Two readers can agree with each other and both be wrong, and kappa measures agreement only (Numiqo).
• Log every BORDERLINE verdict with its controlling criterion. A growing borderline pile on one criterion means the threshold is badly placed, not that cases are getting harder.
• Re-version the rule whenever you change a criterion, and note the date. When you onboard a third reader, you will thank yourself.

There is a structural point behind all of this. When you publish a rule and ask someone else to apply it, you have taken on an obligation to make the rule verifiable, meaning a third person can check whether the rule was followed without asking either reader what they meant. That is the central lesson of the study scoring companies against their voluntary commitments: voluntary measures fail when nobody discloses, in a verifiable way, how the commitment is being met (Wang et al., AIES). Your go/no-go rule is a small voluntary commitment with a small audit problem, and a written rule, blind scores, and a log of disagreements solve it.

If the volume grows past two readers, or the rule starts feeding a model that pre-scores cases for human review, the tooling question changes shape: you need a place where scoring sheets, rule versions and case files live together. That is the kind of build a specialist can take on with you through Quanta, and if you want to talk through the design first, a free 30-minute discovery call covers your goals and a tailored recommendation with no obligation.