A Tie-Breaker Prompt That Requires Evidence: Fields and Examples

The skeleton below is sanitized: no company names, no domain specifics. Fill the three bracketed blocks. Keep the output contract exactly as written, because the rejection rule has to be machine-checkable.

You are resolving a close call. You do not give an opinion.

DECISION
[One sentence: choose Option A or Option B, or return "UNDECIDED".]

OPTION A: [short description]
OPTION B: [short description]

EVIDENCE RULES
1. For each option, list every piece of evidence that bears on the
   decision. For each item give: source id, exact location
   (file and line, section and clause, record id and field),
   and a verbatim quote of at most 40 words.
2. Do not paraphrase in the evidence block. Copy exactly.
3. If a fact you want to use has no source id in the context,
   you must omit it and say "not available in context".
4. If either option has zero evidence items, return UNDECIDED
   with reason "no evidence for <option>".
5. A verdict is valid only if it names at least one evidence item
   for the chosen option and at least one for the rejected option.

OUTPUT FORMAT (reasoning first, verdict last)
EVIDENCE_A: [list]
EVIDENCE_B: [list]
DISCRIMINATOR: [the single evidence item that decides it,
               and one sentence saying why]
VERDICT: [A | B | UNDECIDED]
CONFIDENCE: [0-100]

If you intend to return A or B and DISCRIMINATOR is empty,
you must return UNDECIDED instead.

Two structural choices matter. First, the reasoning fields sit before the verdict. Autoregressive models generate left to right, so asking for rationale and sources ahead of the answer makes the model think before committing rather than justify after, a pattern recommended in prompt engineering best practices. Second, the format uses named fields rather than prose. Galtea's template guidance treats explicit criteria and structured outputs as the baseline for a judge prompt.

Note that the skeleton does not ask for a citation count or a citation quality score. Both invite padding: the model adds weak items to seem thorough. One discriminator is harder to fake than five supporting quotes.

Setup: a pull request adds a second helper, parse_amount_v2, next to an existing parse_amount. The reviewer asks the model which one should be kept.

Bad output: "Both functions look similar. parse_amount_v2 is more readable, so keep it and deprecate the other." No line, no test, no reason tied to behavior.

Good output, in the contract:

EVIDENCE_A ("parse_amount"):
  - src/money.py L41-L47: "return Decimal(raw.replace(',', ''))"
    cited by tests/test_money.py L12 test_european_thousands
  - CHANGELOG entry id=chg-2291: "parse_amount handles grouped digits"
EVIDENCE_B ("parse_amount_v2"):
  - src/money.py L88-L96: "raise ValueError('comma grouping unsupported')"
DISCRIMINATOR: tests/test_money.py L12 exercises comma-grouped input
  and passes against parse_amount only. The existing test suite is the
  deciding evidence; the readability claim is not checkable.
VERDICT: A
CONFIDENCE: 82

The test name is the discriminator, not taste. That follows the verification-first posture in Augment Code's guide to reviewing AI-generated code: the compiler, the diff and the test suite act as referee, and human review degrades past 400 lines reviewed at once, per the SmartBear and Cisco study cited there. Keeping code review honest also means treating the reviewer as its own component: the adversarial code review approach separates the agent that writes a change from the agent that certifies it, and an evidence contract is what makes that separation auditable.

If the pull request also touches manifests or lock files, add one more evidence rule: any changed dependency must be cited with the dependency review result, since that check reports added, removed and updated packages plus known vulnerabilities per GitHub's dependency review documentation.

Setup: one clause says a refund is available within 30 days of purchase; another says a refund is available "while the subscription is active". A customer cancels on day 45 of an annual plan. The question is which clause governs.

The prompt body stays the same; only the evidence rules get a domain line: "quote clause text verbatim, and if a clause contains a precedence or supersession statement, quote that too." Then:

The weak output is exactly what the rejection rule is for. It borrowed the authority of a section number without quoting it, so a human cannot check it. The strong output quotes all three clauses and names section 4.4 as the discriminator. If 4.4 had not existed in the context, rule 3 forces "not available in context", and the correct verdict becomes UNDECIDED with a note to retrieve the precedence clause.

Two extra rules are worth adding for policy work. Treat the scope or applicability clause as mandatory evidence; a refund clause that only applies to regulated products is not evidence for anything else. And cap the quote length, as the skeleton does at 40 words, so long clauses cannot be cited by volume.

Setup: an annotator labeled a support ticket "billing" and a second labeled it "account access". The model is asked to break the tie against a written label definition.

Here the evidence is not prose, it is a passage plus a rule:

EVIDENCE_A ("billing"):
  - ticket t-88421 field=body: "invoice shows two charges for March"
  - label-guide sec-2: "billing: any dispute about an amount charged"
EVIDENCE_B ("account access"):
  - ticket t-88421 field=body: "cannot log in to view the invoice"
DISCRIMINATOR: The disputed charge is the subject of the request;
  the login problem is the means by which it was noticed. Label-guide
  sec-2 matches the subject, and sec-5 ("use the subject of the
  request when symptoms and subject differ") resolves the overlap.
VERDICT: A
CONFIDENCE: 74

Notice what the model is not allowed to do: prefer the majority, prefer the first annotator, or reason from ticket length. The label guide, not the model, arbitrates. Research on models as evaluators that write and apply their own rubrics reports that scoring reliability degrades in factual and knowledge-intensive settings, and that agreement is higher within a model than across models, per GER-Eval. Supplying the guide as retrieved evidence is the practical hedge.

This example is also the cheapest place to measure the prompt. Take 50 historic ties where a senior human already ruled. Run the prompt. If the model returns A or B with an empty discriminator more than a few times, tighten rule 5 and rerun.

A prompt is a request; a check is a guarantee. Separate the writer of the verdict from the code that gates its release, as this piece on forcing an LLM to cite its evidence argues: a citation guard at the exit point checks every cited identifier against the evidence store and aborts the run if any identifier was not retrieved in that run. The prompt persuades; the guard decides.

A minimal guard, runnable wherever the judge output is parsed:

import re

FIELD = re.compile(r"^EVIDENCE_(A|B):\s*(.*)$", re.M)

def first_rejection(output, valid_ids):
    ids = set(valid_ids)
    cited = set()
    has_evidence = {"A": False, "B": False}
    for label, body in FIELD.findall(output):
        if body.strip():
            has_evidence[label] = True
        cited.update(re.findall(r"\b[a-z]{2,5}-\d{2,5}(?:\.\d)?\b", body))
    if not all(has_evidence.values()):
        return "missing evidence block for one option"
    phantom = cited - ids
    if phantom:
        return "uncited source: " + ", ".join(sorted(phantom))
    disc = re.search(r"^DISCRIMINATOR:\s*(.+)$", output, re.M)
    if not disc or not disc.group(1).strip():
        return "empty discriminator"
    if "VERDICT: UNDECIDED" not in output and not cited:
        return "verdict without citation"
    return None

Run it on every candidate verdict and treat a non-None return as a hard failure, not a warning. On rejection, the fallback is UNDECIDED, and a human or a retrieval step takes over. Three practical settings:

1. Route rejections back as a retry that must add the missing evidence, capped at one retry. After that, escalate.
2. Log rejection reasons by type (missing block, phantom source, empty discriminator) so you can tell whether the prompt is failing or the retrieval is.
3. Version the label guide or rubric alongside the prompt. A verdict is only reproducible against the document it cited.

The discipline behind all three is the same one Guidance on verifying agent output describes: distinguish a finding, a fix and evidence, and inspect the evidence rather than trusting the summary. Arize's evaluator guide adds the operational part: validate the judge against reviewed examples before you rely on its scores, and keep the evaluator version visible when comparing results over time.

Use this table to decide whether a given tie needs the contract or a simple prompt.

If two or more cells land in the right column, write the contract. A related decision: whether to judge in one pass or pairwise. Evidently AI's guide notes that judges handle both pairwise comparison and direct scoring, and that judge quality depends on the prompt and the task. Pairwise framing is a tie-breaker's natural shape, but either way the evidence fields are what make it checkable.

One more rule for teams: if the cited evidence is retrieved by a tool, scope that tool tightly. A code review agent with broad credentials is a privileged component, and secret scanning before content reaches the model is a baseline control, as Redfox Security's build guide sets out. An evidence contract that quotes source lines also means those lines left your repository, so the retrieval boundary is part of the design.