What to Do with Interview Evidence You Decide to Discard

Four categories cover almost every legitimate exclusion, and each has a different evidentiary footprint. Naming the category before you decide keeps the decision honest.

Irrelevance. The interview answered a question your study is not asking. This is common in discovery work where an interview drifts into adjacent territory, or where a participant was recruited against a looser screener than the current study uses. The I-Corps guidance on customer discovery interviews makes the underlying point well: interviews aimed at validating assumptions rather than surfacing what the customer actually experiences produce weak material, and weak material can legitimately be set aside. Irrelevance is the easiest category to abuse, though, because "off-topic" is a judgment, not an observation. Tie it to a specific research question and a specific quote boundary.

Unreliability. The participant could not recall the events the study depends on, or the interview was conducted so long after the purchase that the timeline is reconstructed rather than remembered. The field report on putting the Jobs to be Done interview into practice is unusually frank about this: the team interviewed customers who had bought too long ago and ended up with transcripts that were useless for their purposes. That is a clean unreliability exclusion, and it should be logged with the purchase date and the interview date so the pattern is visible across the dataset rather than discovered one transcript at a time.

Duplication. A second or third interview with the same participant, or with a participant from an already saturated segment, adds no new information. This overlaps with what the CASRAI material on data saturation and information power calls a saturation judgment, and the important discipline is to log the judgment and its basis, not just the exclusion. Excluding a duplicate without recording what it duplicated leaves a reviewer unable to check whether the duplication was real.

Legal or ethical restriction. Consent was withdrawn, the participant asked for material to be removed, the transcript contains identifiers that cannot be redacted without destroying the content, or a regulator or contract limits how the data can be used. The literature on data sharing in qualitative research documents how often consent terms and confidentiality promises constrain reuse in ways researchers do not anticipate. Exclusions in this category are non-negotiable, but they still need a log entry, because a future analyst who finds the material sitting in the archive will otherwise assume it is fair game.

One useful decision rule cuts across all four: an exclusion is justified when you can point to a property of the evidence that a second researcher could verify. "The participant bought 26 months before the interview, outside our 18-month window" is verifiable. "I did not find this interview persuasive" is not, and it belongs in a memo, not in an exclusion log.

The Gotranscript audit trail template lists data exclusion decisions among the records to retain, alongside negative case memos, which is a useful reminder that the two belong in the same mental bucket: both are cases that did not fit, and both are most valuable when the reasoning survives.

An exclusion log is a table, not a document. Six fields carry almost all the value.

1. Evidence ID. The stable identifier used everywhere else in the project (participant code plus session number, never a name). The CASRAI checklist insists on stable identifiers for exactly this reason, and using the same ID in the log and in the archive is what makes the link checkable.
2. Date of decision. The day the decision was made, not the day it was written up. If you cannot establish the date, the Atlas guide on audit trails and decision logs recommends saying so explicitly rather than inventing a plausible timeline.
3. Decision maker. One named person. A committee is not a decision maker, and a log that names a committee cannot be queried when the decision is challenged.
4. Reason code. One of the six in the table above, or a project-specific equivalent. Codes should be a closed list, because free text defeats the point of a log you want to count.
5. Linked decision. The analysis, theme, persona, roadmap item, or report section the evidence would have informed. This field is the one most often skipped and the most useful later, because it tells a future reader what changes if the exclusion is reversed.
6. Note. One or two sentences. Verifiable facts and the alternative considered, not a justification essay.

A minimal schema, if you are keeping the log as structured data rather than a spreadsheet:

exclusion:
  evidence_id: P-014-S2
  decided_on: 2026-03-04
  decided_by: lead_researcher
  reason_code: UNRELIABLE-RECALL
  linked_decision: persona-v2-pricing-trigger
  note: >
    Participant purchased 26 months before interview, outside the 18-month
    window agreed for this study. Timeline was reconstructed from email
    rather than recalled. Considered keeping for pricing-trigger signals;
    rejected because the trigger dates are unreliable.
  minority_note: null
  review_point: persona-v2 sign-off

The review_point field is a small addition with outsized value. It says when the exclusion should be looked at again, which is usually the next time the linked decision is revisited. Without it, exclusions accumulate as permanent, unexamined facts about the dataset.

If you are building tooling around this, two habits pay off. First, keep the log append-only: corrections are new rows that reference the row they supersede, never edits in place. Second, generate a checksum or version stamp for the log whenever it is exported, so a report can state which version of the exclusion record it was built against. Teams that already run structured extraction against transcripts can use the same infrastructure to populate candidate log entries for review, which is a natural place for a single API such as Quanta Inference to sit if the rest of your pipeline already passes model calls through one endpoint.

The most common failure in exclusion practice is not a missing log. It is a log that exists while the excluded material sits in the same folder as everything else, unflagged, waiting for the next analyst or the next retrieval step to pick it up.

The fix is to treat exclusion as a property of the artifact, not a note in a separate document. Mark the file itself. A prefix convention works: EXCLUDED-P-014-S2-transcript.md. A metadata field works better: status: excluded, exclusion_id: EX-031. Do both if your archive is browsed by humans as well as searched by tools. The CASRAI guidance on separating identifiers from data and restricting access to the participant key points in the same direction: the archive should carry enough structure that the wrong material cannot be pulled in by accident.

Then enforce it at the point of retrieval. Any query that assembles transcripts for coding, theme development, or synthesis should filter on status rather than on folder location. If your analysis runs as code, make the filter non-optional:

def load_for_analysis(evidence_ids):
    rows = fetch(evidence_ids)
    live = [r for r in rows if r["status"] == "active"]
    blocked = [r["evidence_id"] for r in rows if r["status"] == "excluded"]
    if blocked:
        raise RuntimeError(f"Excluded evidence requested: {blocked}")
    return live

Raising rather than silently filtering is deliberate. A silent filter trains people to ignore it. An error forces the analyst to make a new, logged decision to reactivate the evidence, which is exactly the gate you want.

This matters more now that analysis pipelines include automated reviewers. The work on measuring alignment between human and LLM judges makes the point that a judge is only trustworthy once you know how consistently humans apply the same criteria, and that reference labels should come from adjudication or a true majority vote with ties and low-consensus cases preserved rather than assigned an arbitrary label. Excluded evidence is the same kind of asset: it is a preserved low-consensus case. If it silently returns to the evaluation set, your judge calibration is being computed against material your team already decided not to trust.

There is a lifecycle dimension too. The Galileo guide to calibrating LLM judges in production warns that a judge can agree closely with reviewers at launch and then drift, with dashboards staying green while the judge quietly stops measuring what it was supposed to measure. An exclusion log gives you a tripwire: if review of a fresh sample keeps flagging material that your log says was excluded, either the exclusion was wrong or the pipeline is leaking. Both are worth knowing before the report ships.

Teams disagree about exclusions, and they should. The question is what the record looks like afterward.

The wrong pattern is consensus by attrition. One reviewer wants to exclude an interview, another wants to keep it, the discussion happens in a call, and the final log entry says simply "excluded" with no hint that anyone objected. Six months later the analyst who objected has left, and the record now implies a unanimity that never existed.

The right pattern is a majority decision with a preserved minority note. Record the majority position in the main fields, then add a short note that names the dissenting reviewer, states their position in one or two sentences, and says what evidence would change their mind. Do not delete it, do not paraphrase it into nothing, and do not mark it resolved. A dissent is a live hypothesis about the data, and the minority note is where it lives until the hypothesis is tested.

This is close to what the Arize material recommends for annotation more broadly: collect multiple human judgments, use the disagreements to sharpen the criteria, and preserve ties and low-consensus cases rather than collapsing them into an arbitrary label. For exclusions specifically, there is an extra benefit. A cluster of minority notes on the same reason code is a signal that the code itself is doing too much work. If five reviewers dissent on OUT-OF-SCOPE exclusions in the same study, the research questions were probably too narrow or the screener too loose, and the log is telling you so.

A short checklist for the disagreement case:

• Name the majority decision maker and the date, in the same six fields as any other entry.
• Add the minority note as a field on the same entry, not a separate document that will drift out of sync.
• Set a review point that forces the team to revisit the dissent if the linked decision changes.
• Never let an exclusion be reversed without a new, dated entry that references the original.

The reversal entry deserves emphasis. Evidence that comes back should not simply reappear with its old status restored. A new entry, dated, naming who reversed the decision and why, closes the loop and leaves the trail readable as a sequence rather than a set of contradictions.

Suppose a team runs twelve discovery interviews for a pricing decision. Three transcripts end up excluded. Here is what the log looks like and what it changes.

P-003 is excluded as UNRELIABLE-RECALL because the participant bought 22 months before the interview. P-007 is excluded as OUT-OF-SCOPE because the conversation never left procurement process and the study is about pricing triggers. P-011 is excluded as DUPLICATE because it is a second session with a participant already represented by P-002, and the segment was saturated at that point.

The live set is nine interviews, and the pricing decision is linked to all three exclusions. Now consider what happens when a reviewer asks, in the decision meeting, whether nine is enough. Without the log, the answer is a number and a shrug. With the log, the answer is specific: nine live interviews, one exclusion for recall distance, one for scope, one for duplication, none for reasons that touch the pricing evidence itself. That is a materially different answer, and it takes about ninety seconds to produce.

The reverse case is more instructive. Suppose a fourth transcript had been excluded informally because its pricing comments contradicted the emerging theme. There is no reason code that fits, and the person who made the call is not sure who it was. That is the situation the log exists to catch. The absence of a defensible code is the tell that the exclusion was a preference, and the linked decision field would have shown exactly how much the pricing theme depended on the removal.

The arithmetic is worth stating because it is the insight most teams miss. Three exclusions from twelve is a quarter of the evidence base. A quarter of the dataset moving from live to excluded without a linked decision is not a methodological detail, it is a different study. Recording the link is what keeps the size of that move visible to the people who act on the findings.