Reviewer Decisions as Training Data

A conflict record needs enough structure that a reviewer who was not present can reproduce the call. In practice that means five fields: the two or more claims that disagree, each tied to its source, the scope the conflict applies within (tenant, product, jurisdiction, time), the question the reviewer actually answered, and the decision itself.

The decision field matters more than it looks. "Use source A" is nearly useless as a label, because a model cannot generalize from it. "Use source A because it is the system of record for pricing and source B is a stale snapshot" carries the rule that generalizes. Research on conflict handling in retrieval pipelines makes the same point from the other direction: knowledge conflicts arise from different causes and require tailored solutions rather than one blanket override in the (D)RAGged taxonomy of conflict types. Capture the conflict type alongside the decision, or later reviewers will not know which rule the label encodes.

A capture checklist that works in practice:

1. Duplicate the raw spans, character for character, into the record. Do not paraphrase.
2. Name the conflict type: ambiguity, staleness, disagreement between authorities, or missing evidence.
3. Record the scope explicitly as data, not prose.
4. Write the decision and the justifying rule as separate fields.
5. Attach reviewer identity and the date, even if you strip them before training.
6. Mark whether the reviewer followed an existing precedence entry, amended one, or created a new one.

That last field is the one most teams omit, and it is the one that makes promotion measurable. Without it you cannot tell whether a new decision confirms the table or contradicts it.

The structured record becomes a training example by adding two things: the input the model will actually see at inference time, and the target output you want it to produce. The input should be assembled by the same code path that assembles live conflicts, so the example never looks cleaner or more organized than reality. If production retrieves five chunks, the example carries five chunks, including the noisy ones.

Keep verbatim text alongside any distilled structure. A controlled ablation that swapped only the stored representation inside one fixed retrieval, rerank and reasoning pipeline found verbatim conversation chunks beating LLM-extracted structured artifacts by 15.9 points on LoCoMo and 22.0 points on LongMemEval-S in the representation ablation study. The study also reports that a union store of chunks plus artifacts matched chunks alone, which is the practical recommendation: extract for filtering and search, but never delete the original span. The same paper gives a blunt figure for constraint questions, where summarization-based memory reaches 14.0 percent exact match against 91.0 percent when verbatim wording is retrieved.

A minimal example record looks like this:

{
  "conflict_id": "c-10432",
  "scope": {"product": "billing", "region": "eu"},
  "claims": [
    {"source_id": "policy-2024-11", "span": "Refunds are issued within 14 days."},
    {"source_id": "faq-page", "span": "Refunds take up to 30 days."}
  ],
  "conflict_type": "authority_disagreement",
  "question": "Which refund window applies to EU billing questions?",
  "decision": "policy-2024-11",
  "rule": "Binding policy overrides marketing FAQ when scopes match.",
  "reviewer": "r-217",
  "precedence_action": "confirmed"
}

Notice that the record holds both spans in full and the rule as a separate string. When you build the prompt, the rule becomes few-shot guidance; when you build the precedence table, the rule becomes a candidate row. One capture serves both consumers, which is the entire point of closing the loop.

Few-shot examples steer model behavior even when the demonstration labels are imperfect, and the format of the demonstrations matters as much as their correctness per the few-shot prompting documentation. That is an argument for consistency over volume early on: ten examples in one rigid schema teach more than a hundred in five formats.

Promotion is a decision with a cost on both sides. Promote too early and automation applies a half-formed rule to conflicts it was never meant to cover. Promote too late and reviewers keep answering the same question forever. Neither error is visible without a measurement.

Treat the conflict corpus like a screening problem. Active learning, where the model selects which record the reviewer sees next and learns from each label, is the established way to make human labeling go further; the ASReview literature reports screening time savings of up to 95 percent and notes that a training set of a single relevant and a single irrelevant record suffices to start the cycle in the active learning explainer. The underlying framework is documented as an open-source, machine-learning-aided screening pipeline in the Nature Machine Intelligence paper. Your conflict queue is the same shape: a large pool, sparse genuine conflicts, and a reviewer whose attention is the scarce resource.

The decision rule I recommend, and the one that has held up in practice, is deliberately conservative:

If any row fails, the conflict stays manual and the record joins a "watch" set. A single dissent resets the counter, because a dissent is information: either the rule is wrong or the scope is too wide. Both need a human.

The worked example matters here. Suppose 40 conflicts in a quarter all concern refund windows. Thirty-one resolve to the binding policy. Nine resolve to the FAQ, and all nine come from a period before the policy was revised and from a single product line with a legacy exception. A naive count promotes the policy rule globally and breaks the legacy line. The scope filter catches it, and the promotion happens for every product except the one with a documented exception. That is the entire value of capturing scope as structured data rather than prose.

Once a rule is promoted, the table needs the same discipline a conflict resolution spec gets in any mature system. Enterprise platforms that resolve conflicting actions state an explicit ordered precedence and document examples for each combination, precisely because ambiguity in the order produces unpredictable behavior as in the Archer conflict resolution documentation. Your table should read the same way: an ordered list, each entry naming the conflict type it governs, the scope it applies within, the rule text and the example IDs that justify it.

Three maintenance habits keep the table honest.

First, every promoted entry keeps a link back to the examples that produced it. When one of those examples is later found to be wrong, you can find every rule that leaned on it.

Second, entries get re-validated when their sources change. A rule justified by a policy that has since been revised is not evidence, it is history. The provenance discipline that retrieval teams apply to claims, checking that a cited span actually contains the asserted fact and flagging when it could be outdated or truncated per this provenance checklist, applies to your table entries as much as to retrieved chunks.

Third, the table is versioned and every automated resolution records which version it used. Without that, you cannot audit why the system answered the way it did three months after the fact.

The honest failure mode of precedence tables is over-promotion: a table so aggressive that new conflicts matching an old pattern get resolved wrongly and silently. Counter it with a sampling policy. Resolve automatically, but route a small random share of auto-resolved conflicts back to a reviewer as an audit. If the audit finds systematic disagreement, demote the entry. A precedence table is not a trophy; it is a hypothesis with a maintenance cost.

Reviewers spend time on conflicts. Automation saves time only on repeats. So the return on the loop is a function of the repeat rate, not the total conflict count.

Take a queue of 1,000 conflicts a quarter, each taking a reviewer 12 minutes, and assume 60 percent are repeats of a rule already in the table. Manual processing costs 200 hours. With promotion in place, the 600 repeats resolve automatically; the 400 genuinely new ones cost 80 hours, and the audit sample adds a small margin on top. The saved time is the interesting number, but the interesting question is what to do with it: the same reviewers now label new conflict types instead of re-answering old ones, which is exactly the active-learning flywheel that makes each labeled example more valuable than the last.

Two cautions on that arithmetic. Repeat rate is not constant, because promoting rules changes which conflicts surface. And a wrongly promoted rule is worse than a slow reviewer, because it fails at machine speed. That is why the promotion thresholds above are conservative and why the audit never goes to zero.

When the loop is running and the volume of examples and model variants grows, the practical bottleneck moves from labeling to routing: which model handles which conflict class, and which examples go into which evaluation set. Running that against several providers through one interface keeps the comparison clean, which is what a single-key endpoint like Quanta Inference is for.