How to Build an AI Reviewer That Asks Your Advisors' Questions

The rubric has four mandatory checks and two advisory ones. Mandatory checks block the comment. Advisory checks get appended as a suggestion without blocking.

Two design rules matter more than the exact wording. First, write each check as a question whose answer is yes or no, and require the model to answer it before it produces a verdict; this is a chain-of-thought structure applied to a rubric (LLM-as-a-judge techniques and prompting methods). Second, number the checks and have the model return the numbers it failed. Numbers are what your application routes on, and they are what the advisor sees in the rejection message, so the reason is never "the system didn't like it."

The expected-behavior check (C3) is the one most rubrics miss and the one that produces the largest quality jump. A comment that names an artifact and an observation but stops there, "the intro doesn't establish the problem," still leaves the author guessing. Adding "the first paragraph should name the customer and the cost of the status quo in the first two sentences" makes it a rewrite instruction.

Structure the prompt in labeled blocks so the instructions, the rubric and the input are never confused. Delimiters and an explicit output format are the structural basics that make model responses consistent enough to parse, and giving the model a role plus a small number of examples is what shapes domain-specific judgment (prompt engineering practices).

<role>
You are a review quality gate for a startup advisory program. You do not
improve comments. You only decide whether a comment meets the program rubric.
</role>

<checks>
C1 (mandatory): Does the comment name a specific artifact, such as a document,
section, slide, paragraph, metric, date or meeting?
C2 (mandatory): Does the comment state what is currently there, by quoting,
paraphrasing or pointing to it precisely?
C3 (mandatory): Does the comment state what good would look like, concretely
enough that the author could tell the difference from the current state?
C4 (mandatory): Does the comment give a next step the author could complete
in one work session?
C5 (advisory): Does the comment explain why the change would help?
C6 (advisory): Does the comment address a single issue rather than a theme?
</checks>

<examples>
[three to four labeled examples, see next section]
</examples>

<rules>
Judge only what is written. Do not infer a missing element from context.
If the author's document is provided, use it only to verify C1 and C2.
Quoting a span for each failed check is required. If you cannot quote, the
check fails.
</rules>

<output_format>
Return JSON only:
{"verdict": "pass" | "rewrite",
 "failed_checks": [{"id": "C3", "quote": "...", "why": "one sentence"}],
 "advisory_notes": ["..."],
 "rewrite_prompt": "one sentence addressed to the advisor, naming only the
 failed checks"}
</output_format>

Three details in that prompt do a lot of work. "Judge only what is written" prevents the model from being generous about a comment that implies a problem without naming it. The quoting requirement forces the model to point at the text, which cuts down on hallucinated failures and gives the advisor evidence. And "rewrite_prompt" keeps the rejection message short and specific instead of pasting the whole analysis back.

Treat the prompt as configuration, not code. Prompt versions are best managed as runtime configuration so you can compare versions across subsets of advisors and roll back without a deploy (the same guide on prompt practice). If you are running the call through one endpoint, Quanta Inference gives you text models behind an OpenAI-compatible API at https://quantaintelligence.ai/inference, which keeps the swap to a different model a one-line change during calibration.

Examples are where generic politeness gets replaced by your actual standard. Use real comments from your advisors, anonymized, with two failures and two passes, and keep them short. Label the outcome and the failed checks, not your opinion.

Failing example. Comment: "The metrics section feels weak. I'd want to see stronger numbers here." Verdict: rewrite. Failed: C3, C4. Why: C3, the comment never says what stronger would look like (for example, a retention definition and a cohort window). C4, the author cannot act on "stronger numbers."

Failing example. Comment: "Not sure this positioning will land with enterprise buyers." Verdict: rewrite. Failed: C1, C2, C4. Why: no artifact named, no observation about the current positioning quoted or paraphrased, no next step.

Passing example. Comment: "Slide 6, the pricing table. Observed: three tiers with no usage limits. Expected: a line stating what happens at overage for each tier. Next step: add the overage rule as a footnote, or cut the third tier." Verdict: pass, with a C5 advisory note that the comment does not say why the overage rule matters to the buyer.

Passing example. Comment: "In the hiring plan, paragraph 2 claims two engineers start in month one, but the schedule on page 9 shows month three. Expected: one date across both. Next step: reconcile and pick which month is right." Verdict: pass. This one is a factual inconsistency, and it passes cleanly because it satisfies all four mandatory checks in three sentences.

Two rules keep examples useful. Keep them under about 60 words each, or the model starts imitating length instead of structure. And rotate examples out when they stop appearing in your logs, since examples that never match real failures teach nothing. If you are adapting an existing rubric rather than writing one, start from a published rubric in your domain and alter it to fit, which is the standard advice for building rubrics from scratch (how to create a rubric).

If your advisors span several domains and the rubric genuinely differs (a legal document review and a growth marketing review are not the same task), one option is a model per line of business rather than one prompt with a dozen exceptions. Quanta's expert-led model lets a specialist take the brief and build the reviewer around the specific domain, which you can start from at https://quantaintelligence.ai/.

Do not set a numeric score threshold. Score thresholds drift and are hard to explain to an advisor. Use a decision rule instead.

The last row matters more than calibrators usually expect. Automated judges are not only noisy; they carry known biases, including rewarding verbosity and preferring text that resembles their own output (a review of rubric-based judging and its biases). A fail-closed default means a broken call costs a few minutes of queue time, not a confusing rejection message to an advisor.

On accuracy, treat any single number with suspicion. One widely cited comparison puts LLM judge agreement with human reviewers at about 85 percent, sometimes above the rate at which two humans agree (LLM-as-a-judge explained). That figure comes from a specific benchmark on a specific task, not from your rubric, so use it only as a reason to measure your own agreement rate. A practical target for a first version: a sample of at least 50 recent comments, each labeled by two humans, with the model's verdict compared to the human majority. Anything below roughly four in five agreement on the failure cases means the rubric wording, not the model, needs work.

A worked example. Say 120 advisor comments arrive in a month. Suppose the gate blocks 26 of them (about 22 percent) and 5 of those blocks are wrong, because the human override approves them. That is a false positive rate near 19 percent on blocked items, which is too high to be invisible: the advisor now has a reason to stop writing comments. Two levers fix it. Loosen one check's wording, usually C3, which over-triggers on comments that say "expected" implicitly. Or narrow the check so it only tests for an explicit comparative word or a named alternative. Re-measure on the same 50-comment sample before shipping.

The override is not a nicety; it is how the rubric learns. Give the advisor one button on the rejection: "Send anyway, with a reason," and require a short note. Three fields on that note are enough.

1. Which check the model flagged.
2. Whether the advisor thinks the flag was correct (yes, no, partially).
3. If no, the reasoning in one sentence.

That record does two things. It gives you a labeled disagreement set for free, and it prevents the gate from becoming a tool advisors work around by sending comments by email instead. Keep the override visible to the author as a small label, so the reader of a comment knows it arrived via override. That is honest and it keeps the standard in view.

Logging is where the rubric improves. The paper on learning expert criteria from inline comments takes exactly this position: expert comments are observations of tacit criteria, and accumulated comments can be distilled into a reusable rubric that is then used for prediction and revision (learning rubrics from inline comments). You do not need the full method to get the benefit. You need a table.

Review that log monthly and answer three questions: which check fails most often (that is your advisors' actual weak spot, and a good topic for the next advisory call), which check gets overridden most often (that is the check whose wording is wrong), and which passing comments the author marked unhelpful (that is a missing check). Add a seventh check only when you can cite three logged examples it would have caught. This is the same loop that keeps graded rubrics honest: criteria come from the outcome you want, and descriptors distinguish levels concretely rather than saying "good" or "adequate" (rubrics as a shared language).

Two governance notes. Tell advisors the gate exists and show them the rubric before the first comment is written; a hidden filter is a broken relationship. And keep the author's own reviewer instructions consistent with it, so the comment an advisor writes and the comment an author expects are graded by the same four checks.

Run the gate in shadow mode for two to four weeks. It scores every comment, hides the verdict from the advisor, and emails you a weekly digest. You will learn two things fast: your false positive rate and which advisors would be blocked most. Use that digest to rewrite the rubric and to send a short note to the two or three advisors who need it, before the gate can visibly reject anything.

Then turn on blocking for mandatory checks only, with a single weekly exception: a comment the advisor explicitly marks "quick note, no action expected." That exception prevents the gate from punishing short acknowledgments, which are legitimate and which a checklist will always flag.

Finally, write the rubric into the advisory agreement itself, next to the confidentiality and IP terms that advisory agreements normally carry, since advisors are typically compensated with a small equity grant and sign an agreement covering confidentiality and IP (advisory board setup and compensation). A one-page rubric in the agreement converts the gate from a surprise into the stated standard, and it gives you something concrete to point to if an advisor's contribution never becomes specific.