How to Compare Two Codebook Versions on the Same Batch

A codebook is an operational document: it lists each code, its definition, and examples of how to apply it, and it is what lets a second analyst code new data the same way, according to Koji's codebook guide. Treat each version as a file with a name and a change log, not as a set of edits held in someone's head.

The Delve guide to building a codebook recommends clear definitions plus examples, and adds non-examples: instances that look close to a code but do not belong. Non-examples are exactly where boundary rules live. When someone proposes "count a complaint as X only if it names a specific feature," that sentence belongs in the exclusion criteria of the new version, alongside the old definition it replaces.

Use a small change record per version:

The "codes untouched" row is the claim you are about to test. Write it down before you run anything, because it becomes the decision rule later: if a flip lands in a theme you listed as untouched, the new version has broken something you did not intend to change.

Version control discipline pays off elsewhere too. An essay on annotation drift recommends treating guidelines like code, attaching a guideline version to every label, and re-labeling a small anchor set after each spec change to quantify its impact. That anchor set is your frozen batch, and the impact measure is the comparison you are about to run.

The experiment has one independent variable: the codebook. Everything else stays put. Same transcripts, same order, same unit boundaries, same model, same parameters, same instructions for everything that is not the definition itself.

If you code with a large language model, be aware that the model is not a neutral instrument. A study of open-source models used on patient interviews found that deductive runs produced more codes than inductive ones, that roughly 45 percent of model-generated codes carried meaningful context, and that 22 to 39 percent were duplicative, as reported in the International Journal of Qualitative Methods. Duplication and thin context matter for your comparison: a code that appears twice in one transcript under version B but once under version A may have been "flipped" by repetition, not by the new rule.

Model behavior also shifts with prompt framing. Work on anchoring bias in large language models found that model answers are sensitive to biased hints and that simple prompt techniques were not enough to remove the effect. In practice this means: do not rewrite the instructions in version B beyond the definition you are testing. If you also "improve" the prompt, you cannot attribute the flips to the codebook.

A minimal run manifest, which you can keep as a plain file next to the results:

batch_id:        batch-01-frozen
transcripts:     48 files, checksum recorded
transcript_form: intelligent
unit:            single speaker turn
codebook_A:      sha256: <hash>
codebook_B:      sha256: <hash>
model_settings:  identical across runs
instructions:    identical except the changed definition
run_A_time:      recorded
run_B_time:      recorded

Repeat each run at least twice if your model has any sampling variance, and report how often a transcript disagrees with itself across repeats. Self-disagreement sets a noise floor: a flip between codebook versions that is no larger than the noise floor is not evidence of anything.

Now the comparison itself. Build a table keyed by transcript and theme, with one row per transcript-theme pair, holding the assignment under version A and under version B. Then compute three things per theme:

1. Count of transcripts coded in A only. The theme lost ground.
2. Count coded in B only. The theme gained ground.
3. Count coded under both. Stable, but check the excerpt boundaries.

The difference between counts 1 and 2 is the net change for the theme; the sum is the gross change, which is what you report. Report both. A theme can have a net change of zero while twenty transcripts moved in and twenty moved out, and that churn is a finding.

Agreement statistics exist for exactly this kind of table. Percent agreement, Cohen's kappa, Krippendorff's alpha, and Gwet's AC1 and AC2 each answer slightly different questions, and the inter-coder agreement review in the American Journal of Qualitative Research walks through when each is appropriate. For a codebook comparison, two cautions apply. First, kappa can collapse when one code dominates a theme, which is common in qualitative data; report raw counts next to any coefficient so readers can see what the number is built on. Second, some researchers argue that agreement statistics are a poor fit when the goal is internal consistency rather than replication across teams, a point raised in the same review of inter-coder agreement practice. The Delve guide to intercoder reliability makes the complementary point: the purpose of the exercise is to show that the coding system is clear and transferable, not to produce a trophy score.

A template for the per-theme table:

Read this table as a reviewer would. Theme 3 is clean. Theme 1 moved exactly where the new rule was aimed, which is what success looks like. Theme 4 moved a little, on a case the clarification covers. Themes 2 and 5 moved in directions nobody predicted, and Theme 5 moved in both directions at once, which usually signals an unstable definition rather than a corrected one.

Counts tell you where change happened. Transcripts tell you what changed. Keep a flip log with one row per transcript-theme pair where the assignment differs, and include the excerpt that triggered the difference.

For each flip, record the transcript identifier, the theme, the assignment under A, the assignment under B, and the exact span of text. Then classify the flip:

• On-target flip. The excerpt contains the pattern the new rule was written to handle. Example: version A coded "they never told me the price until the invoice arrived" as a general complaint; version B's new rule says complaints that name a specific missing disclosure get their own code, so the transcript moves.
• Off-target flip. The excerpt does not involve the new rule at all, yet the assignment moved. This is the red flag.
• Ambiguous flip. The excerpt could go either way and the coders disagree on the classification. Keep these separate rather than forcing a call, because a cluster of ambiguous flips usually means the new definition is not yet operational.

The Koji codebook guide is a useful rule of thumb here: if a definition is missing its exclusion criteria or its example quote, a second coder will read it differently. An off-target flip is often the model reading a vague definition differently, not the model making an error.

Two checks catch most problems before they reach the report:

• Read every on-target flip. Confirm the excerpt actually matches the rule, rather than trusting the assignment. It takes an hour for a batch of fifty; skipping it is how a broken rule gets adopted.
• Read a sample of stable assignments. Pick ten transcript-theme pairs coded under both versions and check them by hand. If some of those are wrong in both versions, your comparison is measuring agreement between two flawed readings, not the effect of the change.

If you are running many versions or need a second pair of eyes at scale, a line of business review is the natural way to get expert judgement on the borderline calls; Quanta Consulting runs a free discovery call where you can describe the batch and the rule you are testing.

The rule is short enough to post next to the results:

• Adopt the new version if every flip is an on-target flip on the boundary cases the rule was written for, and no theme you marked as untouched moved.
• Reject and investigate if any flip lands in an untouched theme, or if on-target and off-target flips are mixed beyond what the change log can explain.
• Revise and rerun if the flips are mostly ambiguous, or if the new definition produces self-disagreement above the noise floor across repeated runs.

Run the rule against the table from the previous section. Theme 1 flips are on-target, so they support adoption. Theme 4's single flip matches the clarification. Themes 2 and 5 have flips that version B's change log does not explain, so this codebook is rejected pending investigation, even though the theme the author cared about moved exactly as intended. That is the whole point of looking per theme: a change that does its job can still damage something else.

A worked number makes the stakes concrete. Suppose the batch has 48 transcripts and the old version coded 9 under Theme 1. The new rule adds 5 and removes 1, so Theme 1 now has 13. If the study reported "9 of 48" in an earlier deliverable, adopting the new version silently changes that figure to 13. Anyone reusing the old number needs to know, so record the change in the codebook's revision note alongside the flip log. This is the audit trail that the inter-coder agreement review describes as part of establishing codebook changes.

When the batch is large or you want to run the comparison for several candidate versions at once, Quanta Inference puts text models behind a single OpenAI-compatible API with prepaid pricing in US dollars, which makes repeated identical runs straightforward to schedule and log.

Keep the report to one page plus a flip log, and lead with the decision:

1. Decision and rule. One sentence: adopted, rejected, or revise, and the exact condition that determined it.
2. Per-theme table. Both versions' assignment counts, gross flips, and net change.
3. Flip log summary. Counts of on-target, off-target, and ambiguous flips, with the transcript identifiers attached.
4. Stability note. Self-disagreement rate across repeated runs, so readers know the noise floor.
5. Revised figures. Any theme totals that change in later deliverables, stated plainly.

A useful sanity check on the report: hand it to someone who did not run the comparison and ask them which theme moved and why. If they cannot answer from the table alone, the table is doing too little work. The collaborative thematic analysis guidance from Delve offers a related tool: a shared space for memos attached to codes and excerpts, which doubles as the audit trail for exactly this kind of version-to-version decision.

One more habit pays for itself: rerun the comparison whenever the boundary rule is edited again. A codebook comparison is a snapshot, and the second edit is the one that tends to break things, because it arrives after everyone has stopped paying attention.