A Second-Pass Agent Reviewer Before the Human Gate

Three conditions must all hold. If any fails, fix the upstream problem instead.

Condition one: the gate is measurably a bottleneck. Not "feels slow." Look at the queue: how many items wait at the human step, and what is the median wait. If agents generate faster than humans clear, the queue grows without bound, and that is the signal. Harness describes the same pattern at the repository level: agents can open dozens of pull requests an hour, each touching hundreds of files, while the review process still assumes human speed in the Harness writeup.

Condition two: enough of the defects are structurally checkable. This is the empirical question, and it deserves a real number rather than a guess. Take the last hundred items your humans rejected and tag each rejection by cause. If most rejections are missing fields, format breaks, policy hits or broken links, a second pass pays off immediately. If most rejections are "the analysis is wrong" or "this conclusion is too aggressive," a second pass buys you little.

Condition three: you can afford to be wrong in both directions. A false Block sends good work back to the agent, which costs a retry cycle. A false Pass sends bad work to a human who may trust the pass and skim. Estimate both costs before you build, and pick a rubric threshold to match.

There is also a capacity argument. Human review degrades with volume: the well known SmartBear and Cisco study, covering reviews across 2,500 reviews and 3.2 million lines of code, found defect detection drops to near zero past 400 lines reviewed at once as summarized in the Augment analysis. A machine reviewer does not replace that capacity, but it can shrink each item down to the part that needs a person.

Break failures into four classes and decide, per class, whether the machine or the human owns it.

The first three classes are where a second pass earns its keep. Checklists work because they are explicit: the VisIt pull request template, for example, spells out items like "the developer updated the release notes" and "the developer added tests that prove the fix is effective," each marked done or struck out as not applicable in the VisIt developer manual. A rubric item that cannot be written as a checkbox with an example is not ready for a machine reviewer.

Dependency changes are the clearest example of a mechanical check with real consequences. GitHub's dependency review surfaces which dependencies were added, removed or updated, how many projects use them, and whether any new version carries a known vulnerability in the GitHub docs. If your agent edits a lockfile, "the dependency diff was reviewed and no new vulnerable version was introduced" is a rubric line a machine reviewer can enforce reliably.

Security is a related but trickier class. Studies cited by Endor Labs indicate over 40% of AI-generated code solutions contain security flaws, with injection flaws and hallucinated dependencies among the common categories in the Endor Labs review. A reviewer prompt can catch the obvious cases, but treat "no secrets, no unvalidated input, no invented package names" as a checklist item that routes to a real scanner rather than as a judgment the model makes alone.

The prompt has four parts and no more: the task the agent was given, the output as it will be delivered, the rubric, and the output contract.

The task. Paste the original instruction verbatim, including any constraints. A reviewer that only sees the output cannot tell whether a missing section was an error or never asked for.

The output. Send exactly what the human would see. If the human reviews a diff, send the diff. If the human reviews a rendered document, send the rendered text. Sending a prettified version creates a gap where defects hide.

The rubric. One line per check, each with a pass condition, a fail condition, and a short example of each. Generic criteria underperform. The empirical work on rubric-based judging finds that LLM judges produce interpretable, task-aware criteria and apply them consistently within a model, but that scoring reliability degrades in factual and knowledge-intensive settings in the GER-Eval study. That is a precise instruction: use the reviewer for structure and policy, not for facts.

A minimal prompt skeleton:

You are a pre-review filter. You do not approve or reject work.
You return a verdict per rubric item and one overall routing decision.

TASK GIVEN TO THE AGENT:
{task_text}

AGENT OUTPUT UNDER REVIEW:
{output_text}

RUBRIC (evaluate each item independently):
R1. All required sections present: Summary, Impact, Rollback.
    PASS example: three headings, all non-empty.
    FAIL example: Summary present, Rollback missing.
R2. Every claim that cites a source links to a URL in the allowlist.
    PASS example: link to docs.example.com/policy (on allowlist).
    FAIL example: link to an unlisted blog, or a bare claim with no link.
R3. No new dependency appears in the lockfile diff without a stated reason.
    PASS example: added "requests 2.x" with a one-line justification.
    FAIL example: added a package with no justification line.

OUTPUT FORMAT:
{"items":[{"id":"R1","verdict":"pass|fail|unclear","evidence":"<quote>"}],
 "routing":"block|pass|escalate"}
If any item is fail, routing must be block. If any item is unclear and none fail, routing must be escalate.

Three prompt rules that follow from the judging literature:

First, make the model quote evidence. The Patronus tutorial on LLM judges describes the components of a working judge as input contextualization, comparison against explicit standards, multi-step reasoning, explanation and score synthesis. Quoting the offending span gives the human something to check in two seconds.

Second, forbid global verdicts like "quality: 7/10." A holistic score invites the model to average away a hard failure. Per-item verdicts with a mechanical rollup rule keep the failure visible.

Third, keep the reviewer's instructions short and its criteria long. Long instruction prose with short criteria gives the model room to invent. Short instructions with long criteria give it nowhere to hide.

If you are running several agents on different content types and want one place to call the reviewer model, Quanta Inference exposes text, image, video, audio and embedding models behind a single OpenAI-compatible key, which keeps the reviewer swap a config change rather than a rewrite.

You cannot trust a reviewer you have not measured. The routine below takes roughly a day and gives you a precision and recall number to argue with.

Step 1: Build a labeled set. Take 60 to 100 real outputs that already passed through the human gate, keeping the human verdict. Then seed each one with a known defect from your taxonomy, at least 20 to 30 seeded items across the classes above. Keep a control slice of clean outputs, unmodified, so you can measure false alarms. If real rejections are available, include them too; seeded defects alone overstate recall because they are cleanly checkable by construction.

Step 2: Run the reviewer blind. Do not tell the reviewer which items are seeded. Record every verdict.

Step 3: Score four numbers. These are the ones that matter:

recall    = defects the reviewer blocked / total seeded defects
precision = real defects among blocked items / total items blocked
false-alarm rate = clean items blocked / total clean items
agreement = items where reviewer routing matched human routing / total items

Step 4: Set the threshold from the numbers, not from taste. A plausible starting contract looks like this: recall at or above 0.85 on the mechanical classes, false-alarm rate below 0.10, and zero blocks on the clean control slice. If it misses, tighten the rubric examples rather than adding prose.

Step 5: Re-run monthly on fresh outputs. Rubrics drift as agent behavior drifts. The rubric-based evaluation literature is explicit that the method requires continuous calibration against human experts to remain reliable in the Adnan Masood survey. A reviewer that was accurate against last quarter's failure modes may be checking for problems nobody has anymore.

A note on the score ceiling. In a benchmark of code review agents, all existing review agents taken together solved only around 40% of the tasks derived from human reviews, and agents often focused on different aspects than the humans did in the c-CRAB paper. Treat a reviewer that catches two thirds of mechanical defects as a real win, and do not expect the review quality ceiling to move.

A second pass fails in four predictable ways.

Subtle factual errors. A reviewer checking format will not notice that a stated figure is wrong if the figure is well formatted and plausible. The rubric study is direct about this: scoring reliability degrades in factual and knowledge-intensive settings in the GER-Eval study. Route factual checks to a retrieval step or a database lookup, not to the reviewer's memory.

Judgment calls. Whether a recommendation is too aggressive, whether a design trades off the right things, whether a change is worth shipping now: these have no ground truth, so no rubric can settle them. Keep them with the human. This is the same boundary Google draws when it says aspects of software design are almost never a pure style issue and should be weighed on underlying principles rather than personal opinion in the review standard.

Reviewer bias. Verbosity bias, position bias and self-preference all appear in the judging literature in the GER-Eval paper. Mitigate structurally: randomize the order items appear in the rubric when re-running calibration, keep the reviewer model different from the generator model where you can, and never let the reviewer see the agent's own justification for its choices, since that primes agreement.

Prompt injection from the content under review. If the agent output contains text that reads like instructions, the reviewer may obey it. Use a reviewer with no tool access, put the output in a clearly delimited block, and state in the prompt that anything inside the block is data, not instruction. Redfox's guidance on reviewing agent security notes that a review agent with access to source code, keys and CI environments must be treated as a privileged component, not an afterthought in the Redfox guide.

One more structural mitigation: give the human reviewer a diff view of the machine's findings. If the reviewer's evidence quotes get truncated or the routing is aggregated, a human reading twenty passes in a row will start approving without reading. Show them the failing item first, always.

Ship it in four stages, and do not skip a stage.

1. Shadow. Run the reviewer on every item but route nothing. A human still decides. Compare reviewer routing against human routing and publish the disagreement rate weekly.
2. Advisory. Show the reviewer's verdict to the human as a comment. Measure whether humans agree, and in particular whether they override Blocks. High override rates mean the rubric is wrong, not the human.
3. Filter. Reviewer Blocks return to the agent automatically; everything else goes to a human queue. Watch the queue length. If it drops, keep going. If it grows, something is escalating too often.
4. Sized routing. Add a length signal from the 400-line finding in the Augment analysis: short, clean items go to a fast lane with lighter attention, and long ones keep full scrutiny. The reviewer's real contribution here is triage, not review.

Keep two alarms live at all times. An alarm on the human override rate for Blocks, so a rubric drift is caught within days. An alarm on the Escalate rate, which should stay in a narrow band; a spike means the reviewer is being asked to judge things it cannot judge.

Finally, hold one plain rule above everything: the human gate stays a gate. When a team starts treating a machine Pass as approval, the second pass has not made review faster, it has removed review. The c-CRAB result, where agents solved only around 40% of human-derived review tasks and often looked at different aspects than humans in the benchmark paper, is the argument for keeping a person at the end of the line. The second pass is there to hand that person a shorter, cleaner stack.

If you are standing this up across several lines of business and want the reviewer plus the surrounding workflow rather than a prompt you maintain yourself, Quanta Experts runs context-aware workers that hand back finished deliverables for a human to approve.