How to Prompt a Model for Disconfirming Evidence

The pattern is a sequencing rule, not a magic phrase. Each pass gets its own session or at least its own clean context, and your hypothesis does not appear in passes one or two.

Pass 1: counts without your frame

Give the model the raw records and a fixed tag list with one-line definitions, and ask only for classification and totals. Do not add sentences like "this is to test our hypothesis that X." Do not add examples that lean one way. Ask for an explicit "unclear" or "does not fit any tag" bucket, because a forced choice hides the records that would otherwise contradict you.

Pass 2: verbatims that break the pattern

After you have the counts, run a second prompt that asks for verbatim quotes that do not match the most common pattern, with the surrounding context and a short note on why each one is unusual. This is the pass that produces the sentences you will remember. It also matches a discovery principle from interview research: ask about specific instances, not generalizations, because "tell me about the last time this happened" produces real information while "how often does this happen" produces noise (the customer discovery lexicon from startups.com).

Pass 3: one claim, attacked directly

Now state exactly one claim and ask the model to assemble the strongest case against it, quoting the records. One claim, one pass. If you bundle five claims, you get five shallow summaries and the model starts balancing them against each other, which is the anchoring problem in a new costume. Multi-persona debate research found that systems presenting diverse perspectives reduced participants' conviction toward their initial beliefs, which supports the general point that structured exposure to opposition moves people more than a single balanced summary (the multi-persona debate study).

A minimal pass-three prompt looks like this:

Here are 412 support records, each with an id.
Claim: "Customers churn because setup takes too long."
Task: quote every record that argues against this claim, or that
explains churn through a cause that is not setup time.
For each quote, give the record id and one line on what it shows.
Then give a count: how many of the 412 records mention setup time
at all, and how many mention any other named cause.
Do not summarize the records that support the claim.

The last instruction matters. If you let the model summarize the supporting evidence, it spends its output budget agreeing with you, and the counterexamples arrive thin.

Before any prompt runs, write a one-page frame: the hypothesis in a single sentence, the tag definitions, and, most importantly, predicted counts. "Setup time" should have a predicted number in it, not a vibe. Something like: "I expect setup time to appear in 40 to 50 percent of churn records, and billing to appear in under 10 percent."

Two reasons to write numbers rather than adjectives. First, a prediction with numbers is falsifiable by the pass-one counts, and the anchoring research says the fix is collecting angles, not reasoning harder inside one. Second, written predictions stop the quiet drift where a 12 percent result gets described as "consistent with our thesis." Write the frame in a separate document, with a timestamp, and do not paste it into passes one or two.

Keep the tag definitions frozen too. If you let the model invent tags on the first pass, you cannot compare counts across passes or across months. A hybrid approach, starting from a fixed set of codes and adding new ones as you work through the data, is common practice in qualitative work, but the additions should be visible in a changelog rather than emerging silently inside one prompt (the qualitative coding tutorial from Grad Coach). The same tutorial warns against fully automated coding software on the grounds that tools cannot read the subtleties of language and context, which is why every automated tag here should be checkable against a quote.

Suppose you have 400 churn survey responses and a hypothesis that slow setup drives churn. Here is the whole procedure with numbers.

Step 0, the frame. Write: "Setup time is the top churn cause, expected in 45 percent of records. Price is second, expected at 20 percent. Billing confusion is a minor cause, under 5 percent." Timestamp it.

Step 1, counts. Pass one returns 400 tagged records: setup 156, price 92, missing feature 61, unclear 48, other 43. That is 39 percent setup, 23 percent price, and billing at 3 percent. Setup is real but below your 45 percent prediction, and price came in above your 20 percent. You have not yet read a single complaint.

Step 2, verbatims. Pass two returns 40 quotes that do not fit the setup pattern. Twenty-two of them mention price in the same breath as a competitor comparison. Six mention that setup was fine but the import from a previous tool failed, which is adjacent to setup but not the same tag. Twelve are complaints about support response time that the tag list did not cover.

Step 3, counterexample. Pass three attacks the claim "customers churn because setup takes too long." It returns 18 quotes where price or a missing integration is the stated reason, including three where the respondent says setup was easy. The output also reports that 61 of the 400 records never mention setup at all but do name a specific missing integration.

Step 4, compare to the frame. Setup: predicted 45, observed 39. Price: predicted 20, observed 23. Billing: predicted under 5, observed 3, so your frame survived there. The finding is not "setup is the problem." The finding is that setup is the largest single tag but smaller than you believed, price is larger, and a missing integration tag you did not have is doing real work. That is a decision-changing result, and it came from ordering, not from a better model.

The same logic applies when you test demand rather than diagnose churn. Fake door testing, presenting a feature entry point that does not exist yet and measuring clicks, is a way to validate demand before building, but a click measures curiosity rather than commitment, and the method needs a clear segment and a transparent reveal to avoid damaging trust (the fake door guide from Userpilot, the painted door guide from Koji). The pass-three counterexample here is asking the model for the reasons a high click rate would overstate demand, not for confirmation that the test worked.

Three habits keep the sequence from degrading into a confirmation machine.

Separate sessions. Run passes one and two in fresh contexts. If you must keep one thread, restate the raw data and tag list at the top of each pass rather than carrying a long conversation in which your hypothesis has already appeared.

Sample the tags, not just the counts. Pull 30 records at random, tag them yourself, and compare to the model's tags. The qualitative LLM study found that roughly 45 percent of model-generated codes provided meaningful context while 22 to 39 percent were duplicative, which is a good reason to audit rather than trust a total (the LLM qualitative analysis study). The same study reported some alignment between model and researcher codes, so the goal is calibration, not replacement.

Watch for polite validation. Interview research calls this out directly: the worst outcome is building something nobody wants after a process that only produced agreement. One rule of thumb from practitioner guides is to always ask at least one question that has the potential to destroy your currently imagined business (the Studio Zao interview tips). Pass three is that question in prompt form. A related tactic from prompt psychology is reverse-coding: ask the opposite question as well as the original, so the model cannot guess the desirable answer (the prompt psychology guide).

Log the prompt, not just the answer. Store the exact tag list, the record count, and the frame timestamp with each output. When someone asks in a month why the numbers changed, the log answers before you have to.

If you are running this across several record types, transcripts, surveys and support tickets, normalizing the calls through one API saves the plumbing work so the discipline gets the attention instead. Quanta Inference puts text, image and embedding models behind one OpenAI-compatible key at quantaintelligence.ai/inference, which is useful when pass one needs a cheap classification model and pass three needs your strongest reasoner.

The three-pass sequence is overkill for some jobs and harmful for others.

The last row matters. Qualitative coding exists partly so another researcher can review what you did, which is what transparency means in that tradition (the qualitative coding tutorial from Grad Coach). If the output feeds a regulated decision, the model's tags are a draft and a person signs.

If you want help deciding which of these situations you are in, a short call to scope the work is cheaper than a quarter of misdirected tagging, and Quanta's AI discovery call at quantaintelligence.ai/calendar is a free 30 minutes on goals and a tailored recommendation.