How to Choose the Second Model When Its Only Job Is Reviewing

Write down, in the order the reviewer will apply them, every check it must perform. Most review jobs decompose into four kinds, and they have very different cost and accuracy profiles.

Two decisions flow from that table.

First, format checks should almost never be a model's job. Structured output features, where you supply a JSON schema and the decoder is constrained to it, remove the entire class of format errors before any model judgment happens structured output with vLLM and Ray Serve. Token filtering does the same at the sampling level Friendli structured output, and schema-constrained responses are the standard way to make agent output composable structured output patterns. If your review rubric is mostly "did it return the right fields," you do not need a second language model at all.

Second, everything that remains is where reviewer quality actually matters: whether a claim is supported, whether a policy applies, whether the tone is right. Give each of those a binary, mechanically checkable definition and a reference the reviewer can cite. A rubric item that reads "the summary is accurate" cannot be measured. "Every number in the summary appears in the source document" can.

Build a small set of primary-model outputs that you have already labeled by hand. Between 100 and 300 items is enough to detect the differences that matter for a binary check, and small enough that one experienced person can label it in a day or two. Include the failures you care about most: cases where the primary model confidently stated something false, broke a policy, or drifted in style.

Then score each candidate reviewer on that set and compute two things: agreement with your labels, and what kind of mistakes it makes. Plain percentage agreement overstates reliability because two raters can agree by chance, which is why chance-corrected measures such as Cohen's kappa exist for two raters and Fleiss' kappa for more inter-rater reliability metrics. Choose the agreement metric to matchyour task and label distribution rather than defaulting to the first one you find IAA metric selection.

A worked example makes the threshold concrete. Suppose your calibration set has 200 items, and 60 of them are true failures that must be caught. Reviewer A clears 51 of the 60 (85 percent recall) and flags 20 of the 140 good outputs as failures (86 percent specificity). Reviewer B clears 57 of 60 (95 percent recall) and flags 42 of 140 (70 percent specificity). If you will send every flagged output to a human arbiter, Reviewer A sends 29 items to humans and lets 9 real failures through. Reviewer B sends 45 items to humans and lets 3 real failures through. Neither is "better" in the abstract: one trades reviewer workload for escaped errors, the other the reverse. That trade-off, not a capability score, is the decision.

Two calibration details that teams skip and regret:

• Measure the reviewer alone. Do not let a retry loop or a second pass inflate its score. What matters is the accuracy of one call, because that is what your cost model assumes.
• Report uncertainty. A point estimate on 200 items carries meaningful error bars. The IAA literature recommends reporting confidence intervals and analyzing disagreement patterns rather than publishing a single number IAA metric selection.

Note also what calibration cannot fix. If two skilled humans disagree about whether an output passes, the rubric is under-specified and no reviewer will be consistent inter-rater reliability in practice. Fix the rubric before you evaluate another model against it.

The reviewer's cost is not its price per token. It is the price per primary output, including the review call, any retry, and the human time spent on escalations. That distinction matters because a cheap reviewer that escalates twice as often can cost more than a pricier one that escalates rarely.

Work out the arithmetic before you run a bake-off. The core relationship is simple:

escalations per day  =  volume per day x flag rate
human hours per day  =  escalations per day x minutes per arbitration / 60
fully loaded cost     =  review calls per day x cost per call
                      + human hours per day x loaded hourly rate

Substitute your own numbers. At 10,000 outputs per day, moving the flag rate from 15 percent to 8 percent removes 700 human reviews daily. At four minutes per review, that is roughly 47 hours of arbitration time per day, which is more than one full-time person's worth of difference. That is usually a bigger number than the token bill.

The same logic applies to latency. A review call that adds several seconds to a user-facing response is a product decision, not an infrastructure detail. Teams should define a latency budget per request type and route work to models that fit inside it rather than picking one model for everything balancing speed, cost and quality. The general discipline is to set the acceptable quality and the deadline first, then minimize cost among candidates that satisfy both constraints cost and performance engineering.

The decision rule that follows: among reviewers that clear your accuracy threshold on the calibration set, pick the one with the lowest total cost per primary output, counting human arbitration. Do not pick the most accurate reviewer if it is slower or more expensive than the pipeline can absorb, because you will end up reviewing only a sample of outputs, and a sampled review is a different, weaker guarantee than the one you designed.

If you would rather not maintain separate integrations for each candidate model while you run this comparison, a single endpoint covering many model families makes the bake-off cheaper to run: Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key.

Every reviewer produces disagreements, and disagreements go to people. The design question is whether that queue can drain.

Human-in-the-loop systems route cases to people by confidence threshold, and the choice between blocking and non-blocking review is architectural: blocking review halts execution for approval before an irreversible action, while non-blocking review queues cases while the system continues and applies the correction before any downstream action human-in-the-loop design patterns. A reviewer that flags 40 percent of outputs needs the non-blocking pattern or a much larger team.

Human-in-the-loop research frames the human as a verification and sign-off layer inside the pipeline rather than an after-the-fact auditor, which is exactly the right mental model for a reviewer model's escalations HITL systematic review. That framing produces two operational rules:

• Cap the queue at what staffed arbiters can clear in a shift, and treat any sustained breach as a signal to retune the rubric or the reviewer, not to ask people to work faster.
• Measure arbiter agreement too. If your two human arbiters disagree on 20 percent of escalated cases, the escalation threshold is doing less work than you think, and the reviewer is being blamed for a rubric problem inter-rater consistency.

The delegation framing helps here as well. Giving a model authority to review is a delegation with defined roles and boundaries, and delegation frameworks call for continuous monitoring and dynamic adjustment rather than a one-time handoff Intelligent AI Delegation. In practice, that means logging every reviewer decision with the model identity, prompt and rubric version attached, so a score change can be attributed to a model change rather than to an unexplained drift. Evaluation APIs are designed around precisely this: versioning the judge alongside the dataset so results stay comparable AI evaluation API patterns. Treat the reviewer configuration as versioned infrastructure, not as a prompt that lives in someone's notes.

1. Write the rubric. Split it into deterministic checks and judgment checks. Move every deterministic check into schema-constrained output or code.
2. Build the calibration set. 100 to 300 hand-labeled primary outputs, weighted toward the failures that cost you most.
3. Shortlist three to five reviewers. Include at least one small, cheap model and at least one from a different family than the primary model, to reduce shared blind spots.
4. Score each one, alone, with structured output enforced. Record agreement, recall on known failures, flag rate and median latency.
5. Compute total cost per primary output using your volume, the measured flag rate, and your loaded arbiter hourly rate.
6. Pick the cheapest and fastest reviewer that clears the accuracy threshold and keeps escalations under the arbiter cap.
7. Re-run the calibration set on a schedule and whenever the primary model or the rubric changes.

Step 7 is the one that gets dropped. A reviewer is calibrated against a particular primary model's error profile. Change the primary model and the reviewer's recall can move without any change to the reviewer itself.

Two useful stopping rules. If no candidate clears the threshold, the rubric is probably the problem, so return to step 1 before buying a more expensive model. If the cheapest candidate clears the threshold comfortably, stop evaluating: additional accuracy that the arbiter queue cannot use is waste.

When the review step turns out to need more than a model call, for example a custom dashboard for arbiters or an internal tool that shows the flagged output next to its source, that is ordinary product work: you can describe the tool you need and have a specialist build it with you at Quanta.