When a Model Should Refuse to Answer from Interview Notes

Most teams write the prompt first and retrofit the schema. Reverse that. The retrieval contract is a data structure, and the refusal rule is a property of that structure.

The source_tier column matters more than it looks. Many note-taking systems store both full verbatim and intelligent verbatim transcripts, where filler words and repetitions are removed but exact phrasing is retained, alongside pure summaries as Ember describes. Only the first two tiers can populate a quote field. A summary row gets an empty quote and a tag saying why, so retrieval can return it for context while the generator is forbidden from citing it.

Speaker identity is not optional either. A transcript labeled "Speaker 1 / Speaker 2" is almost unusable for quote work, and renaming speakers immediately, while you still remember who said what, is basic hygiene according to the same extraction guide. Your speaker_id should be assigned at transcription time, not reconstructed later. TheyDo's notes on speaker mapping cover the processing step if you are piping recordings through a tool.

One more data decision: what counts as "on the question's subject." Do not leave this to embeddings alone. Store topic tags at note-creation time from a fixed list and require the retrieved row to match at least one tag from the question. Embedding similarity returns rows that are topically adjacent but say nothing quotable, and those are precisely the rows that tempt a model into paraphrase.

With the schema in place, the prompt becomes short. Its job is to make the no-evidence path the easy path, and to forbid the paraphrase path explicitly.

You answer only from the quote field of the rows below.

Rules:
1. If at least one row has a non-empty quote that addresses the question's subject,
   answer using only those quotes. Attribute each with speaker_id and note_id.
2. If no row has such a quote, reply exactly:
   "No direct quote in these notes addresses this question. Notes searched: <list of note_id>."
3. Never paraphrase, infer, combine or summarize outside a quote.
4. Never use rows whose source_tier is "summary" as evidence.

Question: {question}
Rows: {rows}

Four details carry the weight.

The fixed string is fixed. Do not let the model compose an apology. A constant sentence is greppable, countable and safe to show a stakeholder. Variations ("I couldn't find anything specific, but...") reintroduce the failure you are trying to remove.

The refusal names the notes searched. This turns every refusal into a scoped statement: these eleven notes do not answer this. A reader can then decide whether to widen the search or run more interviews.

The summary tier is fenced off. Without rule 4, a retrieval step will happily surface a summary row, the model will treat it as evidence and you are back to paraphrase with extra steps.

Attribution is inside the answer, not appended. Requiring speaker_id and note_id on every quoted line means a reviewer can jump to the timestamp and check.

If you are running many note sets and question types through the same pipeline, route them through one endpoint rather than maintaining separate integrations; Quanta Inference exposes text and embedding models behind a single OpenAI-compatible API key, which keeps the retrieval and generation halves of this contract in one place.

Suppose you have 40 interview notes, each about 45 minutes, and a stakeholder asks: "Do people abandon checkout because of shipping cost?"

Your retriever returns 12 rows. Eight carry source_tier = summary, two carry verbatim quotes about delivery speed, and two carry verbatim quotes about price but in the context of subscription fees, not shipping. Under the prompt above, the model returns the fixed refusal and lists the 12 note ids.

That outcome feels unhelpful and is correct. The alternative is a paragraph explaining that "several participants expressed sensitivity to cost, suggesting shipping may be a friction point." Every word of that is plausible, none of it is evidence, and a product decision may follow.

Now change the data, not the prompt. You run six more interviews with a question about checkout. Four produce quotes that contain the words "shipping" and a cost figure. The same retriever returns rows, two now have non-empty quotes matching the shipping tag, and the answer becomes two attributed lines with speaker ids.

The numbers to track are these. If 40 notes produce refusals on 30% of your stakeholder's questions, that is a signal about interview coverage, not model quality. That is the diagnostic value of the rule: refusals become a map of where your research is thin.

Compare that with a paraphrase-tolerant setup. To check whether a paraphrase was grounded, a reviewer has to find the source passage, read the surrounding context, and judge whether the model's wording overstates it. At 30 seconds per check and, say, 60 answers a week, that is half an hour of skilled review that still ends in a judgment call. A refusal takes two seconds to verify: either a quote exists or it does not.

A refusal that nobody records is a refusal you cannot tune. Log it the same way you log an answer.

{
  "event": "refusal",
  "query": "checkout abandonment shipping cost",
  "subject_tags": ["checkout", "shipping", "price"],
  "rows_returned": 12,
  "rows_with_quote": 0,
  "notes_searched": ["n-014", "n-019", "n-027"],
  "prompt_version": "refusal-v3",
  "latency_ms": 812
}

Three fields earn their place. rows_returned against rows_with_quote tells you whether retrieval or note coverage is the bottleneck: many rows and zero quotes means your notes are thin on that subject; few rows means the retriever missed. prompt_version lets you compare refusal rates before and after a prompt change without guessing. subject_tags lets you aggregate: if 60% of refusals cluster on one tag, that tag is your next interview guide.

The logging habit has a second payoff. When a stakeholder asks why the system "couldn't answer," you have a timestamped record of which notes were searched, which is a more defensible answer than a shrug. It is the same principle as keeping the audio file as your legal and ethical record while treating the transcript as a working document per the extraction guide.

Aggregate refusal rate is the single number to put on a dashboard. Watch it in both directions. A rate near zero usually means the prompt is leaking paraphrases. A rate near 100% means either your quote field is being populated badly or your question subjects do not match your topic tags.

Every refusal system has two error types, and they need separate test sets. One test set cannot tell you which direction you are failing in.

Build the false-refusal set mechanically. Take 50 existing quotes at random, write a question each one answers, and record the expected note_id. Any refusal on this set is a bug, and the likely cause is topic-tag mismatch rather than the model. Build the missed-refusal set by hand: 30 questions on subjects you know are not covered, mixed across tags that exist in the corpus. A question about shipping cost in a corpus that discusses subscription price is the hard case, because the rows are adjacent but not quotable.

Do not skip the leaked-paraphrase set. It is the one that catches the most dangerous failure, because a paraphrase built from a summary row reads exactly like a good answer. The GRAB-RAG result is the reason to test here specifically: a fluent, relevant passage supporting an answer pulled small models into answering most of the time, and in the misleading condition 63% of those answers repeated the planted wrong entity verbatim per the benchmark. Your summary rows play the role of the planted passage. They are fluent, they are relevant, and they were written by someone else.

The abstention literature is clear that this is a known hard problem and not a prompt bug you can phrase your way out of. Benchmarks for evidence sufficiency show abstention behavior degrading specifically under conflicting evidence, and the authors of ArXiv's newer survey of prompt-based abstention describe the trade between verifiers that detect conflict and the coverage they cost. The survey of abstention in large language models frames the same point: refusal is a capability that needs its own evaluation, not a side effect of a cautious instruction.

Practical target: on the false-refusal set, aim for refusals under 5%; treat anything above that as a retrieval or tagging defect, not a model defect. On the missed-refusal set, any answer is a defect, full stop. Report both numbers together, always, because an easy way to make one look good is to break the other.

Some questions genuinely are not answerable by a single quote, and it is worth knowing which ones before you promise a stakeholder otherwise. "How many participants mentioned shipping?" is a counting question. "Did sentiment change between the first and second round of interviews?" is a comparison question. Neither has a quote as its answer.

For those, the honest move is a different contract, not a looser one. Counting questions can be answered by the retrieval layer directly: return the count of rows with non-empty quotes matching a tag, with the note ids attached. The generated sentence is then a restatement of a mechanical result, and it can be verified in seconds.

The line to hold is between "which notes say this" and "what do the notes mean." The first is a retrieval question with a checkable answer. The second is analysis, and it belongs to a person or to a tool that is explicit about being analysis. Research synthesis has documented failure modes when interpretation gets logged as raw market feedback, and roadmap decisions built that way inherit confirmation bias rather than customer problems per Ember's guidance.

A second practical boundary: leadership preferences. Stated and revealed preferences diverge, and surveys can mislead when respondents report what they think they should say rather than what they did, as discussions of the say-do gap in preference research point out. Your quotes capture what people said. That is useful and it is not the same as what they did. Do not let a well-attributed quote get promoted to a behavioral claim because it reads cleanly. Keep a separate field for observed behavior, and let the refusal rule apply to it too.

When your team is doing this across several research programs and the retrieval, tagging and logging work is taking more time than the analysis, it is reasonable to hand the pipeline to specialists rather than keep rebuilding it. Quanta Experts work from your context and return finished deliverables, which fits the note-processing side of this contract.