A Rubric for Scoring Interview Transcripts with a Model

The prompt has four jobs: carry the rubric, tell the model the output shape, require a verbatim quote per tag, and name the escape hatch. Here is a compact template.

SYSTEM
You score customer interview transcripts against a fixed rubric.
Return JSON only. For every signal, return:
  signal_id, tag (yes | no | unclear), quote (verbatim, or null), rationale (max 15 words)

RUBRIC
S1 Problem is recent and concrete. Definition: speaker names a specific problem
   they hit in the last 90 days, with an event, not an opinion.
   POSITIVE: "Last month our invoice run broke and we lost a day."
   NEGATIVE: "Invoices are always a pain, I guess."
S2 Named current workaround. ...
(continue for each signal)

RULES
- quote must be copied character for character from the transcript.
- If the evidence is absent, tag unclear and set quote to null.
- Do not infer commitment from politeness or compliments.

Three details do the heavy lifting. First, requiring verbatim quotes turns every tag into something you can verify in seconds, which is how you audit at speed. Second, the "unclear" tag gives the model a way out, and a model with no exit will guess. Third, the rationale capped at fifteen words stops the model from writing miniature essays that hide weak evidence.

Analyzing transcripts after the fact is a legitimate use of a model even where running the interview is not; Fitzpatrick endorses transcripts for locating exact quotes while warning against dumping everything into a knowledge base that muddles biased data with good data Fitzpatrick on AI and interviews. Your rubric is what separates the good rows from the muddy ones.

Score toward the end of the call

Commitment tends to concentrate in the closing minutes. Questions about next steps, introductions, and decision makers are exactly where a real signal appears, and interview guides recommend ending by asking for a warm introduction Desert Pacific I-Corps on interview best practices. If you have timestamps, weight the last third of the transcript, or ask the model to note where in the call each quote appears so your audit can check placement.

One transcript per call is the default and the most expensive option. Batching a dozen transcripts per call into a single request, with an explicit transcript ID on each output row, cuts the number of calls and keeps the rubric overhead amortized. The model reads the rubric once per batch instead of once per transcript.

Three rules keep batching honest. Assign every transcript a stable ID before batching and require the model to echo it on each row, or you will spend an afternoon matching rows back to files. Cap batch size by output length, not input length, because twenty transcripts times seven signals times a quote is a large response. And randomize which batch a transcript lands in, so a systematic error tied to prompt position does not align with a real segment of your interviews.

Batching also makes reruns cheap. When you fix a definition after an audit, rerunning three batches takes minutes, and you can diff the new tags against the old ones to see exactly which rows moved.

This is the step most teams skip, and it is the step that makes the rest defensible. Pick a random sample of transcripts, ideally stratified so rare tags get enough coverage, and tag them yourself by hand without looking at the model output first. Galileo's calibration guide notes that stratified sampling surfaces tail failures that random sampling misses, and that raw accuracy inflates on imbalanced labels Galileo on calibrating LLM judges. A set where 90 percent of rows are "no" will make any model look accurate.

Size the sample so the numbers mean something. For a first pass, 30 transcripts tagged by hand across your signals gives you a readable signal on each row of the rubric, and if you have a second colleague, tag the same 30 independently to measure human agreement first. The Arize guide treats human-human agreement as context for the comparison rather than a ceiling, but if two humans barely agree, no model can be expected to Arize on measuring alignment.

Measure agreement properly

Use a chance-adjusted metric. Percent agreement is easy to read but misleading, and Cohen's kappa is the standard starting point for two annotators with categorical labels Arize on judge alignment. Build a confusion matrix per signal so you can see whether errors are false positives, where the model over-tags enthusiasm, or false negatives, where it misses quiet signals.

The right response to a bad row is not a stronger model. It is a tighter definition. If S6 over-tags, the definition probably accepts any hedge, and it should instead require a hedge attached to a specific purchase or timeline.

Read the disagreements, not the aggregate score. If four of five S1 false positives are the same sentence type, your definition needs a negative example drawn from that sentence type. Disagreement is the input to better criteria, and structured rubrics improve by iterating on the criteria that generated them Snorkel on rubric design.

A worked sequence: hand-tag 30 transcripts, score the model, find kappa of 0.41 on S6. Read the eleven model-yes rows, realize seven are generic hedges. Rewrite S6 to require a hedge about a specific commitment, add two negative examples, rerun the same 30, and re-measure. If kappa rises, the definition got better, not the model. Track each signal's kappa across reruns so you can see which part of the rubric is stable.

Two guardrails keep the loop from drifting. Freeze the hand-tagged sample before you edit definitions, or you will unconsciously retrofit labels to defend the new rubric. And keep an untouched holdout set of 30 more transcripts tagged by hand, scored only once the rubric stabilizes, because provider-side model updates can shift scoring distributions without any warning in your dashboard Galileo on judge drift.

Rubric-based evaluation with a model judge is a hybrid stack: deterministic checks on what you can verify, a model judge for nuance, and human oversight for governance and calibration Adnan Masood on rubric-based evaluations. Treat the model as a fast classifier with a known error profile, not an oracle. When you need more interview-derived evidence rather than more scoring, a free 30-minute AI discovery call is a sensible next step, and if you are wiring this into a pipeline, one OpenAI-compatible key for every model keeps the plumbing simple.