Building an AI Reviewer That Asks Your Advisors' Questions

The goal of this step is ten to twenty atomic questions, each answerable by looking at a document. Vague questions produce vague verdicts.

A three-column worksheet is enough. Column one is the question in the advisor's own words. Column two is what makes it answerable. Column three is what a model should look for.

Three rules keep the list clean. One question per line, with no compound questions joined by "and." Every question phrased so a document either contains the answer or does not. Every pass criterion testable by a reader who knows nothing about your company.

The Eisenhower Matrix is a useful check on scope. Its four quadrants separate work that is urgent and important from work that is neither Asana on the Eisenhower Matrix. Advisor questions that keep recurring on high-stakes documents belong in the reviewer; etiquette questions and scheduling complaints belong in quadrant four and should be deleted from the list entirely. The HBS treatment of the same matrix makes the underlying point: the discipline is to keep learning when pressure pushes toward action HBS on prioritization. Trimming the list is that discipline applied to your own automation.

Cap the first version at twelve questions. A twelve-question check runs in one call and produces a review you will actually read.

Structured prompts with delimiters, numbered steps and an explicitly specified output format are what make a model's response consistent enough to parse LaunchDarkly on prompt engineering. Apply that directly: the checklist goes in a fenced JSON block, the document goes in its own section, and the response format is stated as a schema.

{
  "checks": [
    {
      "id": "buyer",
      "question": "Who is the economic buyer and who signs the contract?",
      "pass": "A named role or title appears for both the buyer and the signer.",
      "fail_if": "Only a segment or market is named, or the two roles are conflated."
    },
    {
      "id": "market_size",
      "question": "Is the market size sourced?",
      "pass": "A named report, publisher and year appear next to the figure.",
      "fail_if": "The figure appears with no source, or the source is unnamed."
    }
  ],
  "verdicts": ["pass", "fail", "unclear"],
  "rules": [
    "Judge only what is written in the document.",
    "If the document answers part of the question, verdict is unclear, not pass.",
    "Quote the section heading or slide number that decided the verdict."
  ]
}

The system prompt wraps that block with a role line and an instruction to return JSON only. A short example of one passing verdict and one failing verdict, drawn from a different document, does more for consistency than any amount of prose instruction; the same guidance recommends three to five examples where consistency matters LaunchDarkly on prompt engineering.

The user message is the document itself. For a deck, export to text or extract slide titles and body copy and prefix each block with its slide number so the model can cite back. For a plan or a release note, send sections with headings intact. Do not summarize the document before sending it: summaries hide the missing citation you are trying to catch.

Two design choices matter more than the wording.

Three verdicts, not a score. A numeric confidence invites arguing with the number. Pass, fail and unclear map to three different actions: nothing, fix, and look at it yourself.

One question per check. Compound questions produce verdicts that are true of half the question, which is indistinguishable from a false pass.

If you want a worked comparison of pass criteria quality, the difference is stark. A criterion like "market size is credible" gives a model nothing to test and produces a pass on any number at all. A criterion like "the slide cites a named report and year" produces a fail on exactly the slides an advisor would flag. Write criteria as if you were briefing a new analyst who has never seen your company.

The mechanics are a single chat completion call. Send the system prompt with the JSON checklist, send the document text as the user message, and parse the response as JSON.

import json, os, requests

checklist = json.load(open("advisor_checks.json"))
system = (
    "You are a pre-send reviewer. Judge the document against the checks "
    "below. Return JSON only, with one object per check: id, verdict, "
    "evidence, and fix. Verdicts are pass, fail or unclear.\n\n"
    + json.dumps(checklist)
)

document = open("deck.txt").read()

response = requests.post(
    "https://api.your-provider.example/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
    json={
        "model": os.environ["MODEL"],
        "messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": document},
        ],
        "response_format": {"type": "json_object"},
        "temperature": 0,
    },
)
results = json.loads(response.json()["choices"][0]["message"]["content"])
flagged = [r for r in results["checks"] if r["verdict"] != "pass"]

Four operational notes. Set temperature to zero so repeated runs on the same document agree. Keep prompts as configuration rather than hardcoded strings, so you can revise a criterion without a code deploy; the same source notes that prompt compression can cut token usage substantially while keeping output quality LaunchDarkly on prompt engineering. Log every request and response so you can audit a bad verdict later. And treat the document as untrusted input: if the deck came from outside your team, a line inside it could try to override your instructions.

Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible key with prepaid US dollar pricing, which keeps this loop to one integration even if you later add a vision model to read slide images directly Quanta Inference.

A check that produces a list nobody acts on is worse than no check, because it trains the team to ignore the output. Route on verdict.

• Fail: the author fixes the document. The fix field should name the smallest edit that would flip the verdict, for example "add publisher and year to the market size figure on slide 4."
• Unclear: the author either makes the answer explicit or marks the check as not applicable for this document type, with a one-line reason. Unclear verdicts that recur are a signal the pass criterion is badly written.
• Pass: no action, but keep the evidence quote. It is what makes the review trustworthy.

Post the results where the document lives, not in a separate inbox. A comment thread on the deck or a checklist attached to the release note beats a report in a channel no one opens. Checklist tools have long made the case that recurring processes belong in a shared, visible list rather than in someone's head or a printed binder Checkli, and the same argument applies here.

Keep the reviewer advisory, not blocking, for the first month. Blocking gates get gamed; advisory ones get read. Move to blocking only for checks that have never produced a disputed verdict.

The value compounds only if the reviewer learns. Track two numbers per check, per quarter: how often the model fails the document, and how often an advisor still raises the same question in a live review after the check ran.

That last row is the one people skip. Dead checks add noise and token cost and dilute attention. Delete anything that has not produced a useful flag in two cycles.

This is also the cleanest way to show advisors their input changed something. Advisory relationships are usually equity-compensated with modest grants and limited time, and the most common failure is an advisor who is never heard from again after onboarding WilmerHale on advisory boards. Sending an advisor a monthly line that reads "your sourcing question fired on four of eleven decks this month and all four were fixed before review" is a better use of their time than another status call, and it makes the next request for their attention easier to justify. Guidance on structuring advisory boards is consistent that the relationship should be tailored to a specific need at a specific stage rather than treated as one-size-fits-all Stage 2 Capital on advisors.

One caution about who owns the loop. The reviewer enforces the criteria; it does not decide which questions matter. Revisit the checklist when the advisor set changes or when the company moves to a new stage, because questions that were once decisive become noise. A useful ritual is to review the checklist itself in the same weekly slot you use for the decision log Weekdone weekly review guide.

Some advisor questions should stay human. Anything that depends on tone, relationship history or a judgment about the person receiving the document is a poor fit for a pass criterion, because there is no text a model can check for. Questions about whether the plan is ambitious enough, whether the founder is the right person to lead the next phase, or whether a partnership makes strategic sense are exactly the kind that advisors exist to answer in conversation.

The dividing line is simple: if two competent readers could disagree about whether a document passes, the check is not ready. Keep those questions in the human review, written down so they get asked consistently, and automate only the ones with a checkable answer.