Writing the Reason Code Taxonomy Before the Model Runs

Pull 50 to 100 decisions already made, with outcomes already settled. Do not sample for balance; pull the messy middle, including declines, holds, manual overrides and cases where a reviewer spent several minutes deciding.

Then write sentences. For each case, write one plain sentence in the reviewer's voice describing why the outcome was what it was. Not a feature contribution, not a score threshold, not a bucket like "risk." A lending example shows the difference: a factor described as "length of credit history, oldest account 11 months" is traceable, while "insufficient credit" is a sample-form string a code map printed for a code whose meaning had changed (Botminds on adverse action notice automation).

Now merge. Sort the sentences and collapse anything that means the same thing. Two tests decide whether two sentences are one code:

• Substitution test. If swapping one sentence for the other would change what the applicant must do next, they are two codes. If not, they are one.
• Evidence test. If both are supported by the same field or rule in the decision record, they are one. If one needs a bureau attribute and the other an internal policy rule, they are two.

Keep the list closed: every case maps to exactly one code, with no "other." If you want "other," you have found a missing code. Catch-alls are where appeals go to die, because a reviewer under time pressure chooses "other" instead of resolving ambiguity and the record then says nothing.

An initial list of 10 to 25 codes is about right. Fewer than 10 usually means you merged across the substitution test. More than 25 usually means you wrote model feature names instead of reasons.

Each entry needs a stable code identifier that is never reused, a plain-language sentence suitable for an external notice, a longer internal definition, the specific evidence fields that support the code, and a pair of near-miss cases where it applies and does not.

The near-miss pair carries the most weight. Definitions get interpreted; contrast pairs get applied. If you cannot construct a near-miss, the code is probably a duplicate of its neighbor.

Keep the internal definition and the external sentence as separate fields. The internal one exists to make reviewers consistent. The external one exists to satisfy notification requirements and be readable by a non-specialist. Versioning them together matters because a code map drifting from a model version is precisely how notice reasons go stale (Floowed on credit decisioning and reason codes).

Store the taxonomy as structured, version-controlled data rather than prose, so a pipeline can validate it and every change appears as a diff. Governance teams get the same properties from architecture decision records managed as code: the record is versioned, reviewed through a pull request, and traceable to the systems it governs (ADR-as-Code).

taxonomy_version: rc-4
owner: credit-policy
review_date: 2027-04-01
codes:
  - id: RC_LENGTH_OF_HISTORY
    external: "The length of your credit history is shorter than our policy requires."
    internal: "Oldest reported tradeline below the age threshold in policy CP-4.2."
    evidence: ["bureau.oldest_tradeline_months", "policy_rule.CP-4.2"]
    near_miss: "No tradeline reported at all maps to RC_NO_HISTORY."
  - id: RC_NO_HISTORY
    external: "We could not find any credit history in the report we used."
    internal: "Bureau response contained zero tradelines."
    evidence: ["bureau.tradeline_count", "bureau.response_status"]

Write the near_miss field before the test, not after the first disagreement. It is what makes the test pass or fail.

Take 30 to 50 fresh cases not used to draft the list. Two reviewers who did not write the taxonomy code each case independently, with no discussion. Compute agreement for the whole set and for each code.

Percent agreement alone misleads, which is why Cohen introduced kappa: it accounts for reviewers agreeing by guessing and ranges from minus one to plus one (McHugh on the kappa statistic). A simple illustration makes the trap concrete: reviewers agreeing on 36 of 50 cases looks like 72 percent raw agreement, but once chance agreement is subtracted the kappa is materially lower (worked kappa example). With a short taxonomy and skewed base rates, chance agreement is high, so raw percentages flatter you.

The 0.7 floor is a convention, not law. One research team documents its own rule: below 0.7 they review the codebook for discrepancies and retest on a fresh subset before splitting remaining work, and at or above 0.7 they proceed (University of Florida guidance on calculating interrater reliability). Because it is a convention, your policy should state the number, where it came from and why it fits your context. A reviewer asking "why 0.7" during an appeal should get a citation to your own approved policy.

Two cautions. Kappa is unreliable on thin data, so test fewer codes with more cases each; one coding tool recommends concentrating on essential, frequently used codes with roughly 15 excerpts each (Dedoose on testing IRR with Cohen's kappa). And kappa measures whether reviewers agree, not whether the code describes the decision correctly, so high kappa on a badly written code means both reviewers are consistently wrong (McHugh).

If the whole-set kappa lands below the floor, do not retrain reviewers first. Look at the per-code table. In most first runs one or two codes account for most disagreements, and they are almost always adjacent pairs your substitution test should have separated. Rewrite, add near-miss pairs, and retest on a new sample. Retesting on the same cases after discussion inflates agreement and teaches you nothing.

An unversioned list cannot be defended, because you cannot show which list was in force when a decision was made. Freezing means four things travel together: the version identifier, a named owner as a role rather than a committee, a review date, and a changelog entry for every change with its reason.

Write the taxonomy version into every decision record alongside the policy and release identifiers. Guidance on decision logs is explicit that storing versioned identifiers for release, model and policy is what lets you explain a historical decision honestly, since an answer acceptable under one policy may be blocked under the next (AI agent decision logs). A notice sent under version three should stay reconstructable after version four ships.

The freeze should behave like an accepted decision record: once accepted it becomes immutable, and a later change is a new record that supersedes the old one rather than an edit in place (ADR-as-Code).

Set the review date against the pace of change in what feeds the list: policy rules, bureau attributes, model versions. A code whose mapping depends on a factor the vendor redefines between releases will drift silently, and the drift stays invisible until someone samples notices or an examiner does (Botminds). If the model can change faster than the review date, gate model releases on revalidating affected codes.

Demand for new codes never stops. Unmanaged, the list grows until reviewers face choices they cannot make consistently and kappa falls.

Three rules do most of the work. Add a code only when a case cannot be honestly mapped to an existing one; a code added because a reviewer prefers its wording is a duplicate. Retire codes by marking them inactive rather than deleting them, because historical records still reference them and a record pointing at a code you cannot look up is worse than useless. Rerun the agreement test after any merge or split, since both move the boundaries reviewers apply.

Watch sentence quality, not just code count. Read each external sentence aloud and ask whether someone who has never seen your system could act on it. The compliance catalog of generic versus specific reasons is essentially a list of failures: "we do not offer credit on the terms applied for" is too broad to help, while naming the specific term that was the problem is actionable (CrossCheck Compliance).

Finally, decide what the reviewer interface does with disagreement. If the list is closed but the interface still offers a free-text box, you have not closed it; you have created an unlogged code. Capture the code identifier, the taxonomy version and the supporting evidence fields, recording verdict and public reasons rather than a summary of hidden reasoning, with references pointing to access-controlled data rather than copying it (AI agent decision logs).

If your sample spans several model providers or mixes text, image and structured inputs, routing those calls through one OpenAI-compatible endpoint keeps the logging shape identical across sources and makes the agreement test easy to automate (Quanta Inference).

Say you draft 14 codes from 80 settled cases, then sample 40 fresh cases that two reviewers code independently.

Three codes account for four of the five disagreements, which is normal and why the per-code table beats the headline number. Note that a 12.5 percent disagreement share does not give you the kappa; the chance correction depends on how codes are distributed, so compute it rather than estimating (McHugh).

The fix is specific. RC_INCOMPLETE_DOCS hides whether the missing item was income proof, an authorization or a statement, and compliance guidance requires an incomplete-application reason to identify the specific information that was missing where the file was otherwise decidable (CFPB section 1002.9). The two history codes need a shared boundary note and examples showing where one ends. Retest on 40 new cases and expect the rewritten codes to stabilize and the whole-set kappa to clear your floor.

What you hold at the end is defensible: the list version in force, the evidence field supporting each code, the test result showing reviewers apply it consistently, and the review date governing the next check.