The Evidence Packet a Reviewer Sees Before an Edit Is Approved

Take a support policy document. The source passage in revision 7 of the doc reads:

Refunds are issued within 14 days of the request.

The proposed change, produced by an assistant that was asked to align the policy with a new contract clause, reads:

Refunds are issued within 7 business days of the request.

The packet the reviewer sees:

{
  "packet_id": "pkt_8841",
  "doc": "support-policy",
  "rev": 7,
  "locator": "sec-4.2/para-3",
  "passage": "Refunds are issued within 14 days of the request.",
  "diff_unified": "- Refunds are issued within 14 days of the request.\n+ Refunds are issued within 7 business days of the request.",
  "match_score": 0.86,
  "score_band": "review-fast",
  "reason": "Contract clause 6.1 sets a 7 business day refund window, so the current 14 day wording conflicts.",
  "sources": ["contract-2024.pdf#clause-6.1"],
  "model": "small-reasoner",
  "tokens": {"passage": 13, "diff": 26, "reason": 21, "system": 180, "total": 240}
}

Notice what the packet does not contain. It does not contain the whole document, because the reviewer does not need it. It does not contain a paragraph of explanation, because the reviewer will skim it and trust the edit less. It does not contain the model's confidence in its own answer, because a self-reported confidence and a retrieval match score measure different things and mixing them confuses the reviewer.

The diff follows the same convention the reviewer already knows from code review: a minus line for the old text, a plus line for the new text, and enough surrounding context to locate it. If the change is a pure insertion, the packet shows the insertion point with one line of context above and below. If the change is a replacement of three sentences, the diff shows all three, because collapsing them would hide the actual edit.

The assembly step is deterministic except for one model call. Pull the passage, render the diff, compute the match score with a retrieval or similarity function, and only then call a small model for the reason. Keep the reason call separate from the scoring call so a scoring regression does not silently change the wording reviewers read.

import json, re

SYSTEM = (
    "You write one sentence explaining why a proposed edit is supported by a source. "
    "Return JSON only: {\"reason\": string}. "
    "Do not exceed 30 words. Do not use hedging words such as maybe or possibly. "
    "Name the source element that supports the edit."
)

def build_packet(passage, proposed, locator, score, source_ref, reasoner):
    diff = f"- {passage}\n+ {proposed}"
    raw = reasoner.complete(
        system=SYSTEM,
        user=f"SOURCE: {source_ref}\nOLD: {passage}\nNEW: {proposed}",
        response_format={"type": "json_object"},
    )
    reason = json.loads(raw)["reason"].strip()
    if not reason or len(reason.split()) > 30:
        raise ValueError("reason failed the length gate")
    return {
        "locator": locator,
        "passage": passage,
        "diff_unified": diff,
        "match_score": round(score, 2),
        "reason": reason,
        "sources": [source_ref],
    }

Two details make this robust in production. First, the model is asked for JSON and the script validates it, because free text from a model drifts in shape and field names change without warning when string parsing is the only check as the Agenta guide to structured outputs points out. Second, the system prompt is delimited and stable, with the variable content last, which is the same structure recommended for consistent parsing in LaunchDarkly's prompt engineering guide and in the Prompting Guide's tips on instructions and separators.

Put the instruction first and the data second, and label each part. That is the advice that survives every model change per the Coursiv cheat sheet's anatomy of a strong prompt. It also keeps the reason sentence short: "Use 2-3 sentences" is measurably better than "keep it short," because the model has a number to hit rather than a vibe to interpret.

Cost lives in four numbers. For the refund example the packet total was 240 tokens: 13 for the passage, 26 for the diff, 21 for the reason, and 180 for the system prompt. Scale that to a document with 400 proposed edits and you are at about 96,000 tokens of reason-generation work, and roughly 75 percent of it is the same system prompt sent 400 times.

That is the cached prefix. Prompt caching keeps the processed keys and values of an identical prefix so the next request skips prefill for it, and on Bedrock a cache read is billed at roughly 10 percent of a standard input token while writing a new cache entry carries a premium of about 125 percent per Karini's explanation of prompt caching. For a batch of 400 packets built in one run, the system prompt is written to cache once and read 399 times, so the dominant cost nearly disappears and the packet total falls toward the sum of passage, diff and reason.

Use the budget table as a planning tool rather than a promise:

Prompt compression can cut token use substantially while holding output quality according to LaunchDarkly's guide, but apply it to the system prompt before you apply it to the passage. The passage is evidence; a compressed passage is a paraphrase, and a paraphrase is exactly what the reviewer does not trust.

The match score is a support number, not a confidence number. It answers one question: how strongly is the proposed change backed by the cited source. Compute it with a retrieval embedding between the proposed change and the source element, or with a similarity function over the passage and the source. Fuzzy matching libraries give you the mechanics: Levenshtein and n-gram measures quantify how far one string sits from another as this survey of fuzzy matching algorithms describes, and the practical guidance is to pick the measure by the kind of error you expect rather than by habit per Data Ladder's fuzzy matching primer.

Then set bands, and let the band decide how much of the packet the reviewer sees first:

The decision rule is simple: move the top band boundary down until median decision time in that band crosses your target, then stop. If reviewers in the 0.85 band take 40 seconds, the packet is missing a field or the reason is vague, not that the threshold is wrong. If they take 4 seconds and later reverse the edit, the threshold is too low.

Keep two counters per band: median time and reversal rate after five days. The pair is the health signal. A threshold that produces fast approvals and near-zero reversals is working. A threshold that produces fast approvals and a rising reversal rate is letting unsupported edits through, and the fix is usually the reason field, not the score.

You cannot tune what you do not measure, and you should not measure every packet. Sample between 5 and 10 percent of approved packets each week, stratified by score band, so the top band is not overrepresented. The sample does not need a reviewer's time to collect: log the packet id, the band, the time between first render and click, and whether the edit was reversed within five days.

select
  score_band,
  count(*) as packets,
  percentile_cont(0.5) within group (order by decision_seconds) as median_seconds,
  avg(case when reversed_within_5d then 1.0 else 0.0 end) as reversal_rate
from packets
where approved_at >= now() - interval '7 days'
group by score_band
order by score_band desc;

Read the result against three checks. Median decision time should rise as the band falls; if it does not, reviewers are ignoring the band and reading everything. Reversal rate should stay flat across bands; if the top band reverses more often than the middle, the score is miscalibrated against the source. And the packet count in the top band should grow over successive weeks if the proposing system is improving; if it does not, the fix is upstream of the reviewer.

A grounded review workflow helps here. Content teams that document brand voice, classify content by risk before drafting, and keep a fixed edit checklist see more consistent quality than teams that rely on intuition at the review step according to Glean's guide to AI content review workflows. The packet is the fixed checklist made concrete for one edit. Journals that suspect AI-written reviews have the same problem from the other side: comments arrive detailed but cite issues that do not exist in the manuscript, and detection is unreliable per the COPE case on suspected AI reviewers. A packet that cites the exact source element makes that failure visible, because a reviewer who claims a problem can point at the passage.

Write down the loop the reviewer should not enter: open the document, search for the phrase, read the surrounding section, decide whether the proposed wording is better, check whether the source supports it, come back to the change, click approve. Each of those steps is a way for the decision to take a minute instead of ten seconds. The packet removes every one of them by putting the passage, the diff, the score and the reason in one card.

There is a second, quieter failure: the reviewer accepts everything because the packet looks like work. Guard against that by asking for one rejection reason from a fixed list (unsupported by source, changes meaning, wrong location, style mismatch) and by reviewing rejected packets monthly against approved ones. If the rejection list is empty for a week, sample harder.

When you need the reason call and the embedding call to go through one endpoint across many models, a single key removes a setup step. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API with prepaid pricing in US dollars: https://quantaintelligence.ai/inference. For teams that want a person to review the packet design and the sampling plan with them, a consulting call covers the score bands and the token budget in one session.