How to Audit a Pricing Interview Transcript Against the Brief

Once the checklist exists, segment the transcript. The boundary is the respondent's answer, not the agent's question. That single rule removes most of the confusion in interview audit work.

The procedure:

1. Walk the turns in order and mark each agent turn as either a brief question, a brief-required probe, or unclassified.
2. Mark each respondent turn that fully answers the preceding question. Close the segment there.
3. Any agent turn between a complete answer and the next brief question is drift, unless the brief explicitly authorizes clarification.
4. Carry forward partial answers. If the respondent answered half the question, the probe that finishes it still belongs to the same brief item.

Why the respondent's answer is the boundary: it means an agent cannot earn credit for a topic just by raising it. It has to also draw the answer out. That matters because the value of an interview method hinges on the probing, not the question list. In a study evaluating AI-led qualitative interviews, a decomposition exercise attributed the richer responses mainly to asking dynamic probing questions rather than to the fixed script Conducting Qualitative Interviews with AI. If probing is where the value sits, probing is where the audit has to look hardest.

A small worked example. The brief's Q4 asks, "What would you expect to pay per seat per month?" The agent asks it and gets "I don't know, depends on the package." The brief requires two probes: why that number, and what it replaces. If the agent asks "Would you say more like fifty or more like a hundred and fifty?" it has skipped the required probes and inserted a range the brief never authorized. Both are scoreable defects, and the tagging pass catches them because those turns map to no brief item.

Drift is any agent behavior not traceable to a brief item. In pricing interviews, four patterns account for most of the damage. They are not equally serious.

Skipped probes. The question is asked and the shallow answer is accepted. Q4 is asked, the respondent says "depends," and the agent moves to Q5. This is the most expensive drift in pricing research because a gut answer with no follow-up carries almost no information. It damages coverage and probing scores.

Leading questions. The agent smuggles a number or an opinion into the question: "Most teams like you land around forty, does that feel right?" This is worse than a skipped probe because it contaminates the answer. The Stanford working paper on anchoring effects shows that an initial anchored value shifts willingness to pay in the direction of the anchor anchoring effects on willingness-to-pay. A pricing interview that anchors early has just measured its own anchor. There is a second, subtler bias: hypothetical pricing questions tend to inflate stated willingness to pay, which is why structured formats were developed. One comparison in a meta study cites a public-good estimate three times higher under a hypothetical question than in a simulated-payment experiment the Van Westendorp method explained. De-biased direct question approaches exist precisely because open-ended direct questions are biased a de-biased direct question approach. A leading question stacks anchoring on top of that baseline hypothetical bias.

Unasked numeric anchors. The brief requires a specific anchor (a rate card, a sum of category spend, a competitor price) and the transcript reaches the end without a single number. This is a fail regardless of how well the conversation flowed. If your team runs Van Westendorp or Gabor-Granger work downstream, an anchor-free transcript cannot contribute to it SurveyKing on when each method fits.

Topic substitution. The agent replaces the assigned value question with something adjacent. The brief asks what the respondent would stop paying for if the feature disappeared; the agent asks which features they like most. That is a different measurement and it is not in the analysis plan.

One useful guardrail when you write the rubric: ask whether the agent asked the question in a way that would have produced a defensible answer even if the respondent had been instructed to be unhelpful. That test separates a polite chat from research.

A four-part rubric that is easy to explain and hard to game:

Give each part equal weight only if the brief says so. In pricing work, a sensible default weighting is coverage 30, wording 25, probing 20, anchoring 25. Justify the weights once and keep them fixed across the sample, otherwise transcripts are not comparable.

A worked example with numbers. Suppose one transcript scores coverage 2 (Q4 and Q7 merged, Q9 skipped), wording 3 (one paraphrase of the anchor question), probing 1 (Q2 and Q5 asked without follow-up), anchoring 1 (no number elicited before the agent offered a range). Weighted score is 0.3×2 + 0.25×3 + 0.2×1 + 0.25×1 = 0.6 + 0.75 + 0.2 + 0.25 = 1.80 out of 4. For three respondents, the weighted scores came out at 1.80, 2.30 and 3.10, giving a mean of 2.40. That average is the headline. The table of component means is the diagnosis. Now note the deeper problem: those three interviews left different questions unanswered, so the pricing evidence you have is a patchwork rather than a consistent set.

Decision rule for the final report: publish a mean score, a table of component means and a list of the brief items never covered across the sample. The third list is the one that changes behavior. If an item was never covered in five interviews, the fix is usually in the brief or the question design, not just in the agent.

Aggregation is where audits go wrong. Three rules keep it honest.

Score transcripts blind when possible. Strip respondent names and any agent self-report before scoring. If the agent logs its own assessment of how it followed the guide, do not read it until after.

Compare against the brief, not against other transcripts. Your rubric is the standard. Other transcripts are only useful to spot patterns. The moment an auditor starts scoring "better than the last one," the scale drifts.

Separate specification failures from execution failures. If an item was skipped in eight of ten transcripts, the specification or the question design is probably the cause. If it was skipped in two, the agent is the cause. Report these separately, because the remedies differ.

For teams that run these audits at volume, the biggest cost is tagging. Model choice matters less than consistency: one tagging pass with strict rules on the respondent-answer boundary beats three inconsistent passes. If you are comparing tagging runs or trying different models as the classifier, running them through a single API and logging the labels is the cheap way to spot inconsistency. Teams that prefer a single key across text, audio transcription and image models tend to reach for an aggregator for that, and Quanta Inference is built for exactly that job, with one OpenAI-compatible key and prepaid USD billing: https://quantaintelligence.ai/inference.

If the audits exist because the whole pricing study is being rebuilt, that is a different piece of work, and Quanta will build the site, tooling or document plan with a specialist: https://quantaintelligence.ai/.

Document the values with their source. Every number in a pricing report should carry where it was read: a figure the respondent quoted, an industry reference the brief supplied, or a modeled output. An anchoring audit that cannot trace a number back to its source is not finished. The de-biased direct question work is useful here because it shows how the same willingness to pay can look different under different question structures the ScienceDirect article.

A small template for the per-interview scorecard, written in plain text rather than a table so you can paste it into a tracker:

Transcript: R-014
Brief version: v3
Coverage: 2   Wording: 3   Probing: 1   Anchoring: 1
Weighted: 1.80
Skipped: Q7 (summed budget), Q9 (renewal trigger)
Leading: Q4 probe offered "forty" before answer
Anchor: none elicited in Q4-Q8
Remedy: agent prompt fix, brief unchanged

That template forces the auditor to name the specific items rather than just leave a number. It is also the artifact a sales team can act on, which is the point of the whole exercise the product marketing alliance template.