The Evidence Packet a Reviewer Sees Before Approving

The order matters as much as the contents. A reviewer reads downward and stops when they have enough to decide. Put the elements in the order the decision is made, and most approvals end after the third block.

The vertical arrangement is deliberate: question at the top, quote directly above its source chunk, score as a small badge next to the chunk heading. The eye then travels one short path: what was asked, what we propose to say, where it came from, how strong the match was. Any layout that separates the quote from its chunk, for example a quote in a side panel and sources in a list below, forces the reviewer to match them by hand.

Keep the adjacent context. A quote shown alone invites the reviewer to read intent into a fragment. Showing the sentence before and after, greyed out and non-selectable, costs two lines and removes most "I need to open the document" clicks.

On the score: show the threshold too. A score of 0.71 means one thing when the cutoff is 0.70 and something else when the cutoff is 0.55. A compact form such as score 0.71 / cutoff 0.70 is enough.

<section class="approval" aria-label="Approve AI answer">
  <header>
    <p class="label">Question asked</p>
    <blockquote class="question">How long do customers have to request a refund on an annual plan?</blockquote>
    <p class="meta">Ticket #48213 · Action: send reply to customer</p>
  </header>

  <section class="quote">
    <p class="label">Proposed answer (quoted text)</p>
    <p class="context">...refunds are handled by the billing team within the stated window...</p>
    <mark>Annual plans can be refunded within 30 days of the charge date.</mark>
    <p class="context">...after that period, credits may be issued at our discretion...</p>
  </section>

  <section class="source">
    <p class="label">
      Source: Billing Policy.pdf, section 4.2, p. 7
      <span class="badge" title="Similarity score / retrieval cutoff">0.81 / 0.70</span>
    </p>
    <p class="chunk">4.2 Refund windows. <mark>Annual plans can be refunded within 30 days of the charge date.</mark> Monthly plans are non-refundable after the first billing cycle...</p>
  </section>

  <footer>
    <button>Approve and send</button>
    <button>Approve with edits</button>
    <button>Reject (reason required)</button>
  </footer>
</section>

The markup is plain on purpose. The hard part is not the component, it is the data behind it: the quote span inside the chunk, the document locator, and the score all have to be carried from retrieval through generation to the review record.

A generated answer usually stitches several chunks together. If the reviewer is shown the whole answer and a list of sources, they have to guess which sentence came from which source. Store the mapping at generation time and hand it to the screen.

The practical way is to require the generator to return spans, not just text: for each cited sentence, the chunk identifier and the character offsets of the quoted span within that chunk. Then the UI can render the chunk with the span highlighted and nothing has to be matched by string search at review time. String matching fails on smart quotes, hyphenation and ellipses, and each failure looks like a citation problem to the reviewer, which slows the decision.

This matters because citation failure is not only about invented sources. Research on long form retrieval grounded generation separates the case where a model cites a source that does not exist from the case where it cites a real source that fails to support the claim, and both make a correct looking statement unverifiable (FACTUM, arXiv). A screen that shows the chunk, with the quoted span highlighted inside it, is the cheapest defense against the second case. A screen that shows only a document title and a link defends against neither.

There is a second reason to keep the chunk visible even when the quote is short. Reviewers catch more problems in the surrounding text than in the quote: a policy that changed, a "not" that changes scope, an exception in the next sentence. The chunk is the unit of evidence; the quote is the unit of approval.

Measure the median time from the screen rendering to the approve or reject click. Segment it by action type, because a customer facing reply and an internal note should not share a target. The rule of thumb is simple: routine approvals should land near or under 30 seconds, and anything that regularly runs past that points to a missing element or a hunt.

Worked example. Suppose a team reviews 400 answers a day. If the median approval takes 75 seconds, that is 8.3 hours of reviewer time per day, spread across several people. Cut the median to 25 seconds with a single screen and the same volume takes 2.8 hours, a saving of about 5.5 hours a day. At two approvals a minute, one reviewer covers roughly 100 items an hour, so the queue capacity changes shape, not just the cost.

The measurement is only useful if you can tell which of the four elements was missing. Log, per approval, whether the reviewer opened the source document, whether they expanded a collapsed panel, and whether they used the edit control. Each of those is a symptom:

• Source document opened on most approvals: the location detail is too thin, or the chunk is too short.
• Panel expanded to see the question: the header is collapsing it or truncating it.
• Edits on a large share of approvals: the quote is being used as a draft, and the reviewer is doing generation work. Look at the prompt, not the screen.
• Very fast rejects: the retrieval is surfacing marginal chunks, and the score badge is doing its job. Check the threshold.

Human review of retrieved answers is a rationing problem as much as a labeling one: automated metrics tell you where to look, and review time should go to the items most likely to be wrong (Label Studio). Time to decision is the cost side of that rationing, and it belongs on the same dashboard as the queue.

Not every action deserves the same screen. Approval gates are usually reserved for actions that are irreversible, costly, regulated or high blast radius, and the reverse also holds: read only work such as summaries and retrieval can run without a gate (StackAI). That split lets you spend design effort on the gates that matter.

A workable three tier check:

The distinction between technically authorized and contextually authorized is the reason the gate exists at all: a credential or scope grant made weeks earlier answers whether an agent may call a tool, not whether this specific invocation should run with these arguments right now (Scalekit). For answers, the equivalent is that a model is allowed to draft anything, but a person confirms this quote, from this chunk, at this score, before the customer sees it. That confirmation is only meaningful if the reviewer can see the four elements.

Two operational details that belong on the same screen and are often forgotten. First, the credential or session valid at the pause is not guaranteed to be valid at the resume, which is where many approval implementations break (Scalekit). If approval can sit for an hour, re-check that the action can still run when the human clicks. Second, store the rejected reasons as structured values, not free text only. A reason field with "quote does not answer question," "wrong policy version," "excerpt missing context" and an open field feeds retrieval tuning directly.

Reviewers also need an approve with edits path. Forcing a rejection and a rebuild because one word is wrong throws away the good part of the draft and teaches reviewers to avoid the queue. Let them fix the quote, keep the source chunk attached, and log that the approval was edited.

Run this against the current review page. Each line is a yes or no, and each no is a likely cause of a slow median.

• The original question is visible in full, verbatim, above everything else.
• The quote being approved is visually distinct from the surrounding draft and cannot be confused with the model's commentary.
• The source chunk is shown with document name, section heading and page, line or timestamp.
• The quoted span is highlighted inside the chunk, and the preceding and following sentences are visible.
• The similarity score and the retrieval cutoff appear as a small badge next to the chunk.
• Approve, approve with edits and reject are reachable without scrolling or a mouse.
• The source document opens in a new view at the right page if the reviewer wants it, but nothing requires it.
• Every approval writes an audit record: who, when, question, quote, chunk identifier, score, decision.
• The median time to decision is logged and reviewed weekly by action type.

If two or more items are missing, fix the screen before adding reviewers. More people in a slow queue raises cost without raising quality.

For teams building the rest of the pipeline behind this screen, the retrieval and generation side is where the four elements come from: chunks with stable identifiers, scores carried through the prompt, and spans mapped back. That work is standard retrieval augmented generation practice, and vendor write ups of production deployments emphasize grounded answers with citations as the reason the approach reduces invented claims (Dust). A quiet failure mode to watch: a retrieval system can return a relevant looking document with a high score and still not support the specific claim, which is why the reviewer sees the chunk rather than a citation label (Evidently AI). Improving accuracy, relevance and completeness of answers against a baseline is a measurable program, and the review screen is where those judgments are made one item at a time (Glean).

If you are assembling the pipeline and the review experience together, Quanta Inference (https://quantaintelligence.ai/inference) puts embedding, reranking and text models behind one OpenAI compatible API key, which keeps the score and the chunk identifiers consistent from retrieval to the screen.