How to Audit an LLM's Theme Coding Against a Human Sample

You cannot code everything by hand, and you should not try. You need a sample that supports a claim like "for documents of this type, the model's code assignment matches a trained human often enough to count."

Stratify on the variable most likely to change model behavior: length band, source channel, customer segment, language, or time period. Allocate roughly in proportion to each stratum's share of the corpus, with a floor so small but important strata still get a look. A workable starter budget for a corpus of several thousand documents is 60 to 120 units per stratum, enough to see gross drift and cheap enough to double-code.

Sample at the coding-unit level if your counts will be unit-level counts. If the final statistic is "percentage of documents mentioning theme X," then documents are the units and every document in the sample must be coded completely. Mixing the two is the most common source of a number that looks wrong later.

Keep traceability. Every sampled unit needs a stable identifier back to the source file, speaker, and line or timestamp, because you will revisit specific disagreements while writing decision rules. Affinity mapping workflows solve the same problem by tagging each note with a participant ID plus a timestamp so a theme can be traced to the exact quote GoTranscript's affinity mapping workflow.

A model-human agreement figure is meaningless alone. If two trained reviewers agree on a theme 78 percent of the time, a model hitting 72 percent is near the practical ceiling. A model hitting 94 percent against a human ceiling of 78 percent is suspicious, and usually means the humans were sloppy or the prompt was tuned on the wrong rows.

So run the human baseline first. Two reviewers, independent, no discussion, no shared notes beyond the codebook. Then compute human-to-human agreement. Adjudicate disagreements with a third person or by consensus, and log every adjudication as a new decision rule. That log is the most valuable artifact of the exercise, because it names which codes are genuinely hard.

The guidelines literature is blunt here. When an LLM replaces human judgment in tasks such as coding interview transcripts, the replacement must be justified and grounded in inter-model and model-to-human agreement, and high agreement between models can reflect shared model bias rather than valid annotation, so a human-validated sample is still required LLM Guidelines for Software Engineering. That is exactly why the human baseline comes first: it is the only reference that is not itself a model.

The same source describes a useful loop for stabilizing the human reference: put objects in groups small enough to judge in one to three hours, have two or three experts judge the first group together while writing decision rules, then cycle through independent rating, agreement calculation, discussion of disagreements, and rule updates until the target agreement is reached and held for two or three consecutive groups LLM Guidelines for Software Engineering. A compressed version on 40 units takes an afternoon and saves a week of arguing about model output.

Percent agreement is the plain share of units where two coders chose the same code. It is intuitive and it is flattering when one code dominates. If 85 percent of units are uncoded and the rest carry four themes, two coders choosing "uncoded" almost every time will score above 80 percent agreement while agreeing on almost nothing that matters.

Cohen's kappa corrects for that. It exists precisely because percent agreement cannot account for agreement that would occur by chance, and it ranges from -1 to +1, where 0 means chance-level agreement and 1 means perfect agreement McHugh's overview of the kappa statistic. Kappa is defined for two raters, so it fits the human-human and human-model comparisons here; Fleiss' kappa extends to more than two raters and uses a different calculation, so the two should never be compared directly Quirkos on inter-rater reliability.

Two traps. Kappa collapses when one category dominates, which is normal in theme coding where most units carry no theme. And kappa is sensitive to the number of categories and to sample size, so two decimal places on a small sample implies precision you do not have Quirkos on inter-rater reliability. Round to one decimal place or report a confidence interval.

A compact worked example on 100 sampled units, five codes plus "none."

Three things this tells you that a single accuracy number would not. The human ceiling is 0.71, so the model at 0.58 is 0.13 below it, not 0.13 below perfection. The model is consistent with itself, so the error is a rule mismatch you can name. And the per-theme errors may be large and offsetting even when totals look close, which is why the confusion matrix comes next.

If the inference side needs to run a large sample in one pass with a fixed prompt and stored labels, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the model call reproducible Quanta Inference.

Aggregate agreement hides the only thing you can act on: which codes the model over-applies and which it misses. A confusion matrix with human labels on one axis and model labels on the other shows that at a glance.

Read it in three passes. The diagonal shows codes the model gets right almost always, and those are safe to count. Off-diagonal cells with mass show boundary problems, for example the model assigning "pricing confusion" where the human assigned "onboarding friction"; the fix is a sharper decision rule and one or two examples in the prompt, not a new model. Row and column totals reveal prevalence bias: a code whose column total far exceeds its row total is over-applied, and the reverse means it is missed.

Two drift patterns have different fixes. Boundary drift is systematic confusion between adjacent codes and often costs little in totals because errors offset. Prevalence drift is a bias toward or against a whole category, and it moves your headline percentages, so it is the dangerous one.

Sort each disagreement into one of four buckets and count them: definitional (the codebook was ambiguous), unit-level (the split was wrong), contextual (the human used outside knowledge the model lacked), or genuine model error (the text clearly supports one label). In most first audits, the first two buckets hold the majority, which is a finding about your codebook, not your model.

One check that earns its keep: send the same sample through the model twice with an identical prompt. High self-agreement with low human agreement means the model is confidently wrong, and repeating the prompt will not save you. Low self-agreement means the prompt is underspecified or the unit boundaries are unstable.

Decide in writing, before computing agreement, what counts as good enough. This separates an audit from a vibe check and prevents interpreting results into existence after the fact. Interview practice teaches the same lesson: interviewers drift off track and cannot see it themselves, which is why an observer scores the session against explicit criteria rather than impressions Rob Fitzpatrick on rehearsing interviews.

A rule that works in practice has three parts. First, a floor on human-to-human agreement: if two reviewers land below about 0.6 kappa, stop and fix the codebook, because there is nothing to validate against yet. Second, a model-to-human condition expressed relative to the ceiling, for example model kappa within 0.10 of human kappa and percent agreement within 8 points. Third, a per-theme condition: any theme where the model's recall against the human baseline falls below a set level is flagged not countable and shifts to manual coding, even if the overall numbers pass.

Then declare what happens on each outcome. Pass: count the corpus and report the model numbers with agreement figures and sample size beside them. Partial: count only the themes that passed, label the rest as manually coded, and report both. Fail: do not report model theme frequencies at all; use the model for triage and retrieval and code the full set by hand.

Two guardrails keep the rule honest. Keep the held-out rows untouched until the decision is made, and log the prompt and codebook version next to the agreement result so a later change cannot be presented as if it had been tested. The guidelines for replacing human judgment are explicit that the justification, the measurement instrument, and the agreement evidence all have to be reported, not asserted LLM Guidelines for Software Engineering.

Take 4,000 support conversations and six friction themes. Stratify by channel (chat, email) and customer tier (three levels), giving six strata, allocated proportionally with a floor of 50, which lands at 180 sampled conversations. Two reviewers code each conversation for all six themes, independent and blind, and human-human kappa comes out at 0.69 with 83 percent agreement. The adjudication log produces four new rules about conversations that name a competitor.

Run the model on the same 180 conversations with a prompt containing the codebook, the four rules, and one example per code. Model-to-human kappa is 0.61 and percent agreement 77 percent, within 0.10 of the ceiling, so the aggregate passes. But the per-theme breakdown shows over-applied "pricing confusion" and under-applied "integration limits," and those two fail. You count the four passing themes across all 4,000 conversations and manually code the two flagged themes by using the model as a retrieval filter that surfaces candidates for a human to confirm. The reported result is four automated percentages with agreement figures and sample size, plus two manual percentages. That is a defensible synthesis.

Common failure modes, briefly:

• Coding whole documents when the statistic is per-unit, or the reverse. Fix the unit at design time.
• One human coder, which produces a model-versus-opinion comparison with no ceiling.
• Humans shown model output first, which inflates agreement and destroys the baseline.
• Tuning the prompt on the rows later reported as agreement evidence.
• A single accuracy figure with no base rate; a theme mentioned in 3 of 5 units and 3 of 50 are not the same finding Skimle's guide to quantifying qualitative data.
• Treating high inter-model agreement as validation, when it can simply mean shared bias LLM Guidelines for Software Engineering.
• Confusing reliability with accuracy: two coders can agree consistently and both be wrong Quirkos on inter-rater reliability.