Using an AI Agent to Tag and Cluster Customer Interview Notes

Asking nicely for JSON is the approach that fails in production. A model trained on conversational text will occasionally prepend "Here is the JSON you requested," and a parser that hits that first backtick throws a fatal error, which is how one team ended up with 4,000 unprocessed documents after weeks of clean runs the structured outputs write-up. The reliable answer is constrained decoding, which masks invalid tokens during generation so output conforms to a supplied JSON Schema the JSONSchemaBench paper. If your provider supports a schema argument, use it; if not, keep a validator in the loop.

Your system prompt has four jobs: define the role, list the allowed themes, define the output object, and forbid everything else. Here is a working shape.

{
  "type": "object",
  "required": ["interview_id", "tags"],
  "properties": {
    "interview_id": {"type": "string"},
    "tags": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["theme", "segment_id", "quote", "confidence"],
        "properties": {
          "theme": {"enum": ["onboarding_friction", "pricing_uncertainty", "integration_gap", "support_quality", "switching_cost", "outcome_evidence"]},
          "segment_id": {"type": "string", "pattern": "^INT[0-9]{2}-[0-9]{3}$"},
          "quote": {"type": "string", "maxLength": 240},
          "confidence": {"type": "number", "minimum": 0, "maximum": 1}
        }
      }
    }
  }
}

The system prompt text that goes with it should be blunt: you are a tagging function, you may only use themes from the enum, every quote must be copied verbatim from the segment named in segment_id, and you must return an empty array rather than invent a tag when nothing applies. Constrain quote length so the model cannot paste a whole paragraph under the label "quote". Setting confidence forces a numeric judgment per tag rather than a blanket "high," and it gives you a triage queue: anything below your threshold gets a human read.

Prompting research geared to qualitative work makes the same point about clarity and instruction, that vague requests produce vague output and that instruction prompting, teaching the model the task before asking, beats hoping it infers your intent the prompting for qualitative research session. A fixed enum is the strongest form of that instruction, because it removes the model's freedom to invent categories.

Take twelve 45-minute interviews, roughly 3,600 speaker turns, segmented to about 4,800 segments after splitting long turns. Each interview is one API call: the system prompt plus that interview's numbered segments. A model returns between 8 and 25 tags per interview, so you end up with roughly 200 raw tags before deduplication.

The workflow below is what I would run, and the numbers are the ones you should expect to see.

Validation is cheap and catches most bad output. A short script checks that each quote is a substring of the segment text named by segment_id. In practice a well-constrained model hits near 100% on this check, and the failures cluster around segments where the participant repeated themselves or the transcript tool merged two speakers into one turn, which tells you to fix segmentation rather than the prompt.

Do not ask the model to produce the clusters in the same call. Clustering is a counting problem over the tag table: group by theme, count distinct segment ids per theme per interview, and sort. That gives you a defensible number, such as "switching cost appeared in 11 of 12 interviews across 34 segments," which is the kind of figure that makes a finding publishable. Case study research is explicit that measurable, specific results carry far more weight than vague claims of improvement Weber Associates on case studies, and the tag table is what gives you those numbers without rereading every transcript.

For the quote bank, rank tags by confidence and by how many distinct interviews share the theme. A quote that is the only evidence for a theme is a lead, not a finding. A quote from a segment that three interviews echo is your headline.

If you are running this across several model families or want the same tagging job done with image or audio evidence from the same study, one API key for text, audio and embedding models removes a whole layer of plumbing: Quanta Inference.

This is the step teams skip, and it is the one that decides whether the output is usable. Have a human coder code a random sample of segments blind to the model's tags, ideally one full interview plus a slice of each of the others, say 400 segments. Use the same theme enum. Then compare per segment.

The simplest measure is per-segment label overlap: for each segment, does the human's label set and the model's label set share at least one theme? Report it as a percentage, and report it per theme as well, because aggregate agreement hides the themes where the model is systematically wrong. A second useful measure is Jaccard similarity per segment, the size of the intersection divided by the size of the union, which penalizes a model that sprays four labels where a human applied one. If you want a single chance-corrected figure, Cohen's kappa works when each segment has one label; for multi-label output, report overlap and disagreements instead of forcing a kappa you cannot interpret cleanly.

Expect the first pass to look mediocre. A realistic pattern after one iteration: overall overlap lands in the 60 to 75% range, with two themes doing most of the damage. Typical failure modes:

• Splitting one theme into two. The model tags "pricing_uncertainty" and "switching_cost" for the same complaint about migration effort. Fix by tightening the enum definitions, not by asking the model to merge.
• Interviewer turns tagged as evidence. The model picks up a paraphrase in the interviewer's question because it sounds like a summary. Fix by filtering interviewer segments out of the tag input entirely.
• Confidence inflation. Nearly every tag comes back at 0.9 or above, which makes the threshold useless. Fix by giving the model three anchor examples per confidence band in the system prompt.
• Quiet participants under-tagged. If one interview yields three tags and the rest yield twenty, check whether the participant spoke in long turns that got segmented badly.

The research consensus supports treating the model as an augmenting tool rather than a replacement for qualitative judgment, with human input needed to preserve interpretive depth and rigor the Child Trends case study. The per-segment audit is how you make that consensus operational instead of aspirational.

Then rewrite the taxonomy once. In my experience one revision, usually merging two over-split themes and adding one theme the human coder used that the model had no label for, moves overlap up by 10 to 15 points on a second pass over the same 12 interviews. Run the full set again with the revised enum, re-audit on a fresh sample of segments, and stop when the disagreements you have left are ones a second human coder would also have made.

Once the tags validate, the evidence base is a table, not a document. Each row is one segment-theme pair with the interview id, segment id, verbatim quote, confidence and the human audit result. Everything downstream, from a case study draft to a sales enablement page, pulls from this table.

This matters commercially. Buyers increasingly do their own research before talking to anyone, and peer evidence is what gets a vendor shortlisted case studies and the buying journey. Well over a third of B2B content marketers call case studies an effective tactic, ahead of several other formats the Content Marketing Institute case study tips, and technology marketers report heavy production of customer stories Elliott Pearce on B2B case studies. Almost everyone, though, struggles to collect enough of it: 77% of one vendor's new customers said they lacked sufficient evidence and 64% said they were losing time trying to gather it UserEvidence on collecting customer evidence. A tagged evidence table is the direct answer to that shortage, because one interview yields many reusable, attributable proof points rather than one write-up.

A few rules keep it honest. Keep the interviewer's words out of the quote bank. Never publish a quote whose segment failed validation. Get explicit approval before publishing any quote, as testimonial interview guides routinely require the FreeOnlineSurveys testimonial questions guide. And keep the audit results, because when someone challenges a theme count, the answer is a filtered table, not a memory of what the model said.

If you want the pipeline run for you on a specific line of business, with an expert reading your context and returning finished deliverables, that is what Quanta Experts is for.

The failures are predictable enough to list.

It generalizes. Ask for themes and you get "usability," which no segment supports cleanly. The enum plus a rule that every theme must attach to a verbatim quote blocks most of this.

It agrees with itself. Run the same interview twice and you will get different cluster names and slightly different tag counts, because generation is probabilistic the structured outputs write-up. Constrained decoding fixes the shape of the output, not the choice of label the JSONSchemaBench paper. Pin the temperature low, run the full set in one batch, and treat any theme that appears in one run but not another as unstable.

It loses the negative case. Participants often say what did not happen, and a tagger tuned for positive themes will skip "we never got the integration working." Add explicit themes for blocked and abandoned outcomes.

It happily clusters. Given free rein, the model will name clusters that sound plausible and are wrong. Keep clustering in your own code, where you can show the count and the segment list behind every cluster.

It drifts across a long batch. With a huge taxonomy, later calls apply labels differently from earlier ones. Keep the enum short (six to nine themes), version it, and store the version with every tag.

Practical guardrails on top of constrained decoding: validate against the schema before anything is written, fail closed on invalid output rather than retrying silently, log the prompt and the raw response per interview, and keep confidence thresholds visible to whoever reads the results.

Building the site or app that hosts this evidence base, a searchable quote library, a dashboard of theme counts by interview, is a separate job from the analysis. That is where Quanta fits: describe the tool you need and a specialist builds it with you.