When the Tie-Breaker and the Human Disagree: An Override Log

Here is the mechanism. When a reviewer overrides a model, the system only needs the final decision to proceed. The reasoning is optional for execution and mandatory for learning, so any schema that is driven by execution needs will drop it. Free-text fields get truncated, normalized, or never written because the form treats them as a comment box rather than a required field.

The cost is not abstract. Research on parallel agent systems finds that when two agents run the same task they disagree somewhere between 20% and 40% of the time depending on task type, and that the typical system response is to silently pick one answer, after which the intermediate disagreement vanishes from the logs the tianpan.co analysis of disagreement patterns. Debugging those failures takes three to five times longer than debugging a single-agent failure precisely because you cannot tell which party was wrong or that they disagreed at all. The same arithmetic applies to a model and a human: the disagreeing minority is the signal, and the majority verdict is the noise.

There is a second reason to insist on the reviewer's own words. Confidence scores are not calibrated probabilities, so a system that defers to the most confident party ends up following the most overconfident one the tianpan.co analysis of consensus and arbitration. A reviewer's justification is not a probability either, but it is checkable. "The vendor's certificate expired two weeks before the invoice date" can be verified. "Score 0.61, overturned" cannot.

Keep the schema boring. Boring schemas survive contact with app developers who are under deadline.

Two extra columns pay for themselves almost immediately: reviewed_at and reviewed_by, both nullable until the weekly sample picks the row up. That turns the log into a work queue rather than an archive.

A row might look like this:

{
  "override_id": "ovr_01J9F3K2",
  "rule_id": "invoice-amount-mismatch-v4",
  "rule_snapshot": {"compare": "po_total_vs_invoice_total", "tolerance_pct": 0.5, "on_match": "auto_approve"},
  "model_score": 0.61,
  "threshold": 0.80,
  "score_direction": "higher_means_approve",
  "human_reason": "PO total is 40 percent below invoice because half the shipment was backordered and the PO was amended in the ERP, not in the vendor portal. Vendor attached the amended PO as page 3.",
  "reason_code": "stale_source_of_truth",
  "final_outcome": "approved_partial",
  "reviewed_by": null,
  "reviewed_at": null
}

Note the rule snapshot. Storing the rule body inline is the same discipline behind worker-specific routing in durable workflow engines, where the step that must run on a particular host returns a queue name that subsequent steps use explicitly rather than relying on ambient state Temporal's worker-specific task queues pattern. The override row carries its own context so it stays readable when nothing around it does.

Reviewing every override is the right answer only while volume is low. Once you pass roughly 50 overrides a week, full review collapses into rubber-stamping, and a rubber-stamped review is worse than none because it launders the appearance of oversight.

The decision rule this article recommends:

• Under 20 overrides a week: review every override.
• 20 to 200 a week: sample a fixed 20, stratified. Take 8 at random from the whole population, 6 from the highest-scoring overrides (the model was nearly right, which is where tie-breaker tuning lives), 4 from the lowest-scoring, and 2 flagged as suspicious by anyone on the team. Stratification beats pure random because the extremes are where the rule is wrong.
• Above 200 a week: increase the sample by 10 per additional 100 overrides, capped at 60. Beyond 60 a week, a single reviewer stops reading carefully.

The sample size is fixed in advance, not chosen after looking at the data. If you sample until you find something, you will always find something, and you will tune your rules to noise.

Two operational notes. First, the review must happen against the stated reason, not the outcome. An override that turned out fine can still rest on a reasoning error, and an override that turned out badly can be a correct call made with bad luck. Judge the chain of reasoning the reviewer wrote down. Second, keep the sampled row immutable. Corrections go in an append-only review record, the same way a distributed job queue keeps completed jobs in the table with a terminal status rather than deleting them YugabyteDB's distributed job queue data model. You want to be able to ask what the queue looked like in any past week.

If the log is a table and the review is a calendar reminder, the review will lapse. Make the log a queue with leases.

The pattern is well understood in distributed systems. A lease is time-bound exclusive access to a resource, and the node holding it must refresh it or lose it Martin Fowler's description of the lease pattern. Applied to override review: a reviewer claims a row for a fixed window, and if they do not finish and mark it, the row returns to the pool.

The subtlety is ownership on release. The classic bug in distributed locks is a worker whose lease expired deleting a lock that now belongs to someone else, which is why every release must check that the stored owner token matches before deleting the Go distributed locking writeup. The same discipline applies here:

-- claim up to five unreviewed overrides for this reviewer, for 24 hours
UPDATE overrides o
SET claimed_by = $1,
    claim_expires_at = now() + interval '24 hours'
WHERE o.override_id IN (
  SELECT override_id
  FROM overrides
  WHERE reviewed_at IS NULL
    AND (claimed_by IS NULL OR claim_expires_at < now())
  ORDER BY created_at
  LIMIT 5
  FOR UPDATE SKIP LOCKED
)
RETURNING o.override_id, o.human_reason, o.model_score, o.rule_snapshot;

And the release path must be ownership-checked:

-- mark reviewed only if this reviewer still holds a live claim
UPDATE overrides
SET reviewed_at = now(), review_notes = $3, claim_expires_at = NULL
WHERE override_id = $1
  AND claimed_by = $2
  AND claim_expires_at > now();

If that update affects zero rows, the claim lapsed and someone else may now hold the row. Report it, do not retry blindly, and let the lease expire. This is exactly the ambiguous-outcome case the locking writeup warns about: an error does not prove the write did not happen.

One more decision worth making explicitly: at what count does an override escalate to a second reviewer? A workable rule is any override where the model score was within 0.10 of the threshold, because those are the rows that will drive your next rule change, and getting them wrong is expensive.

This is the step almost everyone skips, and it is the one that makes the log worth its storage cost.

When you change a tie-breaker, write down which override rows motivated the change. A minimal table is enough:

The expected_effect field is what turns this into engineering rather than opinion. "We expect overrides coded stale_source_of_truth to drop by half and overrides coded genuine_dispute to stay flat" is a falsifiable claim, and you can check it against the log on the review date.

The pattern of requiring a linked justification is borrowed from conventions that work in practice. Conventional Branch, for instance, exists so branch names carry machine-readable intent and so automated systems can act on that intent the Conventional Branch specification. The same idea applied to rules: a rule revision with no linked overrides is a revision nobody can audit.

There is a limit worth naming. If your arbitration logic is itself a model, you inherit that model's biases. Benchmarks of LLM judges catalogue position bias, self-preference bias and length bias, with position swaps shifting accuracy by more than ten percentage points in code evaluation tasks Evidently AI's guide to LLM-as-a-judge. If you use a model to triage which overrides get human review, log the model's selection and sample outside it. Research on how language models arbitrate conflicting evidence finds that arbitration behavior is systematic rather than random: models show distinct preferences for text over numbers and follow recency more consistently than explicit reliability cues the arXiv study on evidence arbitration. Any triage model you insert will have a house style, and it will shape which human reasoning you ever see.

Suppose the tie-breaker fires on invoice approval. Model score above 0.80 auto-approves, below 0.50 auto-rejects, and the middle band routes to a reviewer. Over four weeks:

• Week 1: 31 overrides. Under the threshold, so full review. Three rows share the reason code stale_source_of_truth and all three involve a vendor portal that lags the ERP by more than a day.
• Week 2: 44 overrides. Full review. Five more stale_source_of_truth rows, and the reviewers' notes reveal they are all checking the same ERP screen manually.
• Week 3: 58 overrides. Full review, but it takes the reviewer most of a Friday. Six more stale_source_of_truth rows.
• Week 4: 71 overrides. You cross into sampling. You take 20: 8 random, 6 highest-scoring, 4 lowest-scoring, 2 flagged. Fourteen of the twenty contain the same phrase about the ERP screen.

At this point the log has paid for itself. The change record writes itself: new rule, check the ERP amendment timestamp before routing on vendor portal totals. Motioning overrides, fourteen override_id values across weeks one to four. Expected effect: stale_source_of_truth reason codes drop by two thirds and manual ERP lookups stop.

Then the honest part. Run the review date four weeks later. If stale_source_of_truth did not drop, your reasoning was wrong somewhere, and the log tells you where: go back to the motivating rows and read them again. That is the entire loop, and it is only possible because the reviewer's words were stored.

If the volume of overrides is large enough that your reviewer capacity, not your rule quality, is the constraint, an AI expert configured for that line of business can take the first pass on classification while a person keeps the sign-off. That is what Quanta experts are built for, each one reading your context and returning finished deliverables, with the override log as the artifact that keeps the person in charge.