How to Score Advisor Feedback for Specificity Before Acting on It

Score every substantive advisor comment on four yes-or-no questions. One point per "yes," zero for "no." The total runs from 0 to 3, and two elements are required before a comment can score above 1.

The design borrows from how analytic rubrics work. An analytic rubric splits a product into distinct traits and judges each independently rather than giving one overall impression (ASCD). Here the traits are claim, evidence, test and scope, and judging them separately stops a confident delivery from inflating the score of an empty comment.

The rubric also gains credibility from how rubrics perform in general. A review of rubrics in higher education finds positive effects on performance, reduced performance anxiety and higher perceived fairness because criteria are known in advance (Polimi OpenKnowledge). Telling advisors the rubric before a review round costs one email and changes what comes back.

Two scoring conventions keep the scale honest.

First, evidence must be checkable by you without another conversation. An advisor's assertion "I have seen this fail" is a claim about their experience, not evidence about your draft, unless they name the case.

Second, the test element does not require a formal experiment. A test can be a single question to three readers, a search for a counterexample, or a reread of one paragraph with the audience named. What matters is that the comment implies a stopping condition: a result that would settle whether the concern is real.

A worked example, scored:

Advisor: "Section 4's claim that the migration takes a day assumes a clean data model. Our portfolio companies all had messy models, so that number will mislead. Either qualify the claim for clean schemas or add the remediation time."

• Claim: names the one-day migration claim in section 4. Pass, 1.
• Evidence: points to portfolio companies with messy models. Checkable if the advisor names even one. Pass, 1.
• Test: the qualifier itself is testable against your own customer notes. Pass, 1.
• Scope: names the section and the condition (clean schema). Pass, 1.

That comment takes two minutes to revise. Compare it with "the middle is weak, tighten it," which scores zero on all four elements and can absorb an afternoon.

The score is a routing rule, not a grade of the advisor. Each band gets one default action.

Score 2 or 3 (testable concern). Act, or ask one clarifying question about the missing element. If claim and scope are present but evidence and test are not, the single question is: "What would I look at to check this?" If claim and evidence are present but scope is missing, ask: "Which audience or section does this apply to?" One question per comment, not a thread. The scoring itself tells you which question to ask.

Score 1 (partial). The comment has one usable element, usually a named claim. Rewrite the comment in your own words as a hypothesis and put it in the log for the next reader test. Do not ask the advisor anything yet; the missing pieces are probably not in their head, and a clarifying exchange will produce a score-1 answer.

Score 0 (preference or mood). Log it with advisor, date and the exact wording. Do not rewrite. A zero is not an insult. It may reflect a real quality problem the advisor cannot articulate, and the log exists to catch those when three zeros cluster on the same paragraph.

The Harvard Business School treatment of the matrix is useful here because it frames restraint as a managerial capability: the discipline to keep learning when pressure pushes toward action, and to wait until understanding is grounded enough to support durable change (HBS Online). Revising to satisfy a score-0 comment is the writing equivalent of intervening before you understand the problem.

One caution on volume. A reviewer who sends forty comments is not automatically your most valuable advisor. Score them all, then plot count against average score. The advisor with eight comments averaging 2.4 matters more than the one with forty averaging 0.7, and the log will show it.

The clarifying question does two jobs: it extracts the missing rubric element, and it teaches the advisor what a usable comment looks like, without a lecture.

Rules that hold up in practice:

• Ask about exactly one missing element. Multiple questions let the advisor answer the easy one.
• Make the question answerable in one sentence. "What would I see in the draft that would show this is wrong?" is one sentence.
• Attach the answer to the draft. "If you have a minute, could you paste the sentence where it breaks for you into the comment thread?" turns a score-1 note into a score-2 note permanently.
• Do not defend the draft in the same message. Defense and clarification in one thread produce an argument, and the argument produces a score-0 reply.

For a score of 2, the clarifying question is optional because action is already justified. The decision rule: if you can name the test yourself, act and skip the question; if only the advisor can name it, ask. That split keeps you from spending review cycles on confirmations.

The log is a table with one row per comment. Keep it in the same tool the team already uses for review, or as a plain sheet.

Two summary figures come out of the log after a few rounds. First, each advisor's mean score, which is the best available proxy for how much of their time converts into draft changes. Second, the share of comments scoring 2 or 3, which measures the review loop's efficiency as a whole. If that share sits below half, the problem is usually the brief, not the advisors: they were asked for impressions rather than against criteria.

Templates for repeating processes make this lighter than it sounds. A reusable checklist turns a recurring workflow into a standard operating procedure so the same steps run every time, and that structure is exactly what a feedback log needs across rounds (Checklist). The same logic applies to any recurring audit: define the steps once, then follow them without renegotiating (Checkli). A short fixed checklist on the advisor's side ("state the claim, the evidence, the test, the scope") has the same effect on incoming comments.

The rubric pays a second dividend when it is visible to advisors before they read. Sending a three-line brief before each round changes the distribution of incoming comments, and it saves you from scoring the same vague note three times.

A brief that works:

Please review sections 2 and 4 only. For each comment:
1. Quote or name the exact claim you doubt.
2. Say what you are comparing it to (a case, a number, your experience).
3. Say what I could check to find out you are right.
4. Say which audience or condition it applies to.
Comments that cannot answer these are still welcome, but mark them "preference" and I will log rather than act.

The last line matters. It gives the advisor a legitimate channel for taste, which keeps the log honest and removes the incentive to dress up a preference as a fact. Rubric research supports the transparency move: criteria known in advance let the person receiving feedback know what to change and how, rather than discovering the standard after the work is judged (Polimi OpenKnowledge).

For formal review settings, peer review guidance reaches the same conclusion from the other side. Reviewer guidance stresses specific, constructive comments with line references and warns that vague comments add little value, and it recommends noting line numbers during a first read so references are easy to write later (Wiley's reviewer guide). That is the claim-and-scope elements of the rubric, arrived at independently.

There is a limit worth stating. A rubric scores the comment, not the truth. A high-scoring comment can describe a real flaw in a draft that is actually fine, and a low-scoring one can point at a genuine problem the advisor cannot articulate. The score tells you what to do next, not who is right. The log and the reread after each round are what settle truth.

After roughly four rounds, the log supports a decision that gut feel handles badly. Use these thresholds, and treat them as defaults you adjust rather than laws.

• Mean score below 1.0 across two consecutive rounds, with no improvement after the brief was sent: move the advisor to a different role, such as introductions or a single domain question, rather than general review.
• High mean score but very low volume: keep them and ask narrower questions, since they are probably short on time rather than short on insight.
• Many score-3 comments that you consistently refute in the outcome column: keep them, because a well-argued wrong comment is still doing work, and refuting it hardens the draft.

The broader advisory literature supports the underlying idea that fit matters more than prestige. Advisor relationships are not one-size-fits-all and should be tailored to the specific need at the specific stage, with an agreed time commitment and a named owner on the company side (Stage 2 Capital). A review loop is one of those needs, and a name on the loop's owner line is what keeps it running. A good advisor is also a domain expert in something the company needs in the next stretch of work, not a general presence (Matt Munson). Scoring comments is how you find out which expertise is actually reaching the page.

If the loop itself is the bottleneck, and scoring, routing and logging are taking more time than the writing, that is a process worth handing to a tool. Quanta can take a described workflow (score each comment, route by total, log with advisor and date) and build it with you at https://quantaintelligence.ai/.