How to Write a Go/No-Go Rule a Second Reader Can Score Blind

Every scored criterion needs four elements, and all four must be present before scoring begins. Write them in the same order every time so readers can scan.

Three design choices do most of the work.

First, keep scales small. Binary yes/no, or a three-point scale (meets fully, meets partially, does not meet), force a decision. Wide scales with undefined midpoints invite readers to express taste rather than observation, which is exactly the variance you are trying to remove.

Second, name the evidence source and make it retrievable. "Customer said the workflow is painful" is not a source; "the signed interview note dated before the pilot started, in the shared drive" is. Customer discovery guidance is blunt on this point: the goal of an interview is to reconstruct a real event, including what happened next and how the person decided, not to collect general opinions (Alloy Growth Lab). The same standard applies when you turn that interview into a scored criterion.

Third, write an explicit rule for absent evidence. If a criterion cannot be scored, the reader needs a default that is identical for everyone. "Cannot determine counts as not met" is defensible and simple. Silence is not.

Readers get into trouble when the aggregation rule is buried in the criterion descriptions. Put the decision logic in its own short block, above or below the criteria, and write it as arithmetic or as a table of outcomes.

Two common shapes work:

• Gating: every criterion on the critical list must score fully met, and secondary criteria have a minimum total.
• Scored: criteria carry weights, the total maps to a band, and one hard floor overrides the band.

Write the second one as an explicit rule so nobody has to infer it. For example: total 8 or more of 10 is go, 5 to 7 is conditional go with named mitigations, 4 or below is no-go, and any "not met" on a critical criterion forces no-go regardless of total.

The value of separating logic from evidence is that disagreement has two possible causes and you can tell them apart. If two readers agree on every criterion but produce different verdicts, the aggregation rule is unclear. If they disagree on individual criteria, the measurement is unclear. You cannot diagnose either problem if the rule reads like prose.

Structured decision processes elsewhere make the same separation. Stage-gate reviews define the criteria separately from the gate meeting, and each gate produces a fixed set of outcomes such as go, kill, hold, recycle or conditional go, so no one has to interpret what a verdict means (monday.com gate review guide). The FAA's advice to establish committed-to-stop points exists precisely because an ambiguous decision boundary invites late, unsafe improvisation (NBAA).

Pick ten cases that span the range you expect: two you are confident are clear gos, two clear no-gos, and six in the messy middle. The middle cases do the diagnostic work.

The procedure, in order:

1. Freeze the scored rule. Record a checksum or date stamp so both readers use the identical text.
2. Give each reader the same ten case packets, with the annex withheld.
3. Ask each reader to score every criterion independently and record the evidence they used, one line per criterion, with a source location.
4. Forbid discussion until both score sheets are submitted. No side channel, no "quick question."
5. Collect both sheets before anyone sees the other's scores.

Blind means blind in both directions. The second reader must not see the first reader's sheet, and neither reader should see the decision the project team expects.

The evidence line matters more than it looks. When two readers disagree, the recorded sources tell you whether they looked at different documents (a packet problem), read the same document differently (a wording problem), or applied different thresholds (a scale problem). Three different fixes, and the evidence log is the only way to tell which one you need.

If you want a quantitative check beyond your own counts, the kappa statistic is the standard tool for agreement between two raters on categorical judgments, and it corrects for agreement that would happen by chance (McHugh, Biochemia Medica). The tutorial literature is clear about its limits: there is no universal cutoff, and when nearly all cases fall into one category, kappa can be low even when raw agreement is high, a phenomenon usually called the kappa paradox (The Analysis Factor, Numiqo). For a ten-case pilot screen, report percent agreement per criterion first and treat kappa as a secondary number.

Now compute three numbers, and report all three.

Exact-match rate per criterion. For each criterion, count the cases where both readers gave the identical score and divide by ten. A criterion at 10 out of 10 is doing its job. A criterion at 6 out of 10 is a coin flip with extra steps.

Directional gap. Separate the disagreements into over-scoring and under-scoring. If reader B marks "met" where reader A marks "not met" in four of four disagreements, you do not have noise, you have a leniency bias in one reader or an ambiguity that one reader resolves generously.

Verdict-level match. Apply the decision logic to both score sheets. Do both readers reach the same final go, conditional go or no-go on each case? This is the number your stakeholders actually care about.

A worked example shows how fast this diagnoses the rule. Suppose ten criteria across ten cases gives 100 criterion scores per reader. Reader A and Reader B agree on 82. Of the 18 disagreements, 12 sit on two criteria: one asks whether the customer "clearly understands the value," the other asks whether the rollout plan is "realistic." Both use three-point scales with no evidence source. The remaining six disagreements are scattered one each across six well-specified criteria, which is normal reader noise.

The fix is not to retrain the readers. It is to rewrite the two bad criteria. Replace "clearly understands the value" with "the named buyer stated a specific number for time saved per week, quoted in the interview note" and score it yes or no. Replace "realistic" with three separate checks: a named owner, a dated milestone, and a dependency list. Then rescore the same ten cases with fresh readers. If agreement on those two criteria moves from 4 out of 10 to 9 or 10 out of 10, the rule is ready.

Set your threshold before you look at the data, or you will rationalize whatever you get. A defensible starting point: every critical criterion at 9 of 10 or better, verdict-level agreement at 9 of 10 or better, and no criterion below 7 of 10 anywhere. Criteria that fail get rewritten and rescored. Two rewrites without improvement usually means the criterion is not observable from the evidence you have, and it should be dropped or converted into a data-collection task.

This is not a novel idea; it is standard practice anywhere a rubric decides something consequential. Rubric design guidance consistently pushes toward concrete, decidable criteria with descriptions of what each score level looks like, because vague criteria produce inconsistent scores even from experienced graders (Microsoft Education). Research on voluntary corporate commitments makes the same point in a harder setting: when commitments are underspecified, scoring them across organizations produces wildly different results, and the authors had to build a rubric of concrete, decidable indicators to score consistently at all (Wang et al., AIES).

Go/no-go rules rarely score documents alone. They score teams, vendors or partners, and the people being scored can feel the difference between a fair process and an arbitrary one. If your rule includes any criterion about stakeholder support, engagement evidence or team behavior, apply the same observable-test standard there.

Engagement measurement guidance is useful here because it insists on defined targets and repeatable indicators rather than impressions: pick the metric, define how it is counted, and track it the same way each time so you can compare across periods (Simply Stakeholders). A criterion like "stakeholders are engaged" fails the blind test instantly. A criterion like "at least two named stakeholders outside the project attended a review in the last four weeks, per the meeting record" passes.

Readiness assessment frameworks make the same structural move: they break a fuzzy question into pillars and score each one against evidence, so the output is a profile of strengths and gaps rather than a single feeling (Prometheus Agency). Copy the shape. Your go/no-go rule should produce a profile, not a mood.

One caution on scope. The more criteria you add, the more chances for disagreement, so resist the urge to score everything. Criteria that do not change the verdict are decoration. Run a simple check: for each criterion, would a "not met" ever flip the outcome? If not, move it to the annex as context.

Run this list in order. Each item is a stop sign if it fails.

• The scored rule contains no examples, no rationale and no recommendation.
• Every criterion names an observable test, a retrievable evidence source and a fixed scale.
• Every criterion has an explicit rule for missing evidence.
• The decision logic is a separate block with named outcome bands and any hard floors.
• Ten cases are prepared, including at least six borderline ones.
• Two readers scored independently and blind, with evidence logged per criterion.
• Exact-match, directional gap and verdict-level match are computed per criterion.
• Every criterion below threshold has been rewritten and rescored.
• The annex holds the rationale, so the pilot audience still gets the story.

If you are building the scoring sheet, the case packets or the tool that collects the two readers' scores, that is ordinary application work. You can describe what you need and have a specialist build it with you at Quanta. If your rule includes criteria that require reading free-text notes at volume, an inference endpoint is the natural place to standardize that extraction, and Quanta Inference puts text, image and embedding models behind one OpenAI-compatible key so the scoring pipeline does not depend on a single vendor.