How to Write a Decision Rule an Agent Can Apply

You cannot build conditions from prose. Pull the log into a table where each row is one decision and each column is one field. Do it in three passes.

Pass one: name the inputs. Read ten rows and list every fact a decider used. Expect four to eight distinct fields. If you find fifteen, you are logging too much; keep the ones that changed the outcome at least once.

Pass two: normalize the values. Free text kills rules. Map "Gold", "gold tier", "premium" to one value. Map error codes to a fixed enumeration. Anything that stays unique per row (a customer name, a free-text note) cannot be a condition and should be dropped from the rule.

Pass three: label the action. Convert each logged decision to one action from a closed set. If you find yourself wanting an action like "handle appropriately", the log row is not a decision yet.

A worked example from a support queue, where the log has forty rows and the actions are refund, escalate and deny:

condition order_value_usd > 500        -> action escalate
condition error_code in [E12, E19]     -> action refund
condition plan == "enterprise"         -> action escalate
condition retry_count >= 3             -> action refund
default                                -> action deny

Four conditions and a default cover a surprising share of operational queues. The default is not laziness; it is the honest statement that the rule does not know, and it should route to a named human rather than to a guess.

Policy engines show why the separation matters. Open Policy Agent's decision logs record the policy path queried, the input to the query, the result and a decision identifier for traceability per the OPA decision log documentation. That is exactly the shape you want: input in, result out, an identifier you can replay. A rule with the same shape can be tested the same way.

One ordering insight the log gives you that no policy document does: which condition wins when two match. Sort conditions by how often they appear in the log and how costly a wrong action is, put the expensive ones first, and write the order down. Ordering ambiguity is the second most common cause of an agent applying a rule "correctly" to the wrong outcome.

A condition is executable when an agent can evaluate it from data it has at run time, with no judgement. Three tests for each condition:

1. Can the agent read this value from a system of record? If it comes from a human's memory at decision time, it belongs in the escalation path, not the rule.
2. Is the comparison deterministic? "Order value over 500" is. "Customer seems frustrated" is not, and it belongs in a separate classifier with its own tests.
3. Is the field populated on most rows? A condition that is null half the time produces silent defaults.

Actions need the same treatment. An action should name a tool call, a queue or a status change that the agent can perform, plus what it must record. Requiring the agent to write the matched condition and the log row it was derived from turns each production decision into a new test case.

LLM agents need the rule in their own channel. Addy Osmani describes keeping a project-local file such as ai_decisions.log and instructing the agent explicitly to log what, why and how, with regular human verification of the log in his automated decision log write-up. The same pattern works for rules: a rules.md in the repository, with the conditions in a fixed format the agent can quote back when it acts.

There is a reason to keep the rule text short and literal rather than paraphrased by a model. Language models trained on "A is B" do not reliably infer "B is A", a failure documented across model families as the Reversal Curse, with GPT-4 answering one direction of a relation far more often than the reverse in the paper's real-world tests reported in the Reversal Curse paper. A rule phrased as "low margin plus high volume means escalate" may not fire when the agent sees high volume plus low margin. Write conditions in one fixed order and test that order.

RULE refund_v3
Order:
  1. order_value_usd > 500            -> escalate
  2. error_code in {E12, E19}         -> refund
  3. plan == "enterprise"             -> escalate
  4. retry_count >= 3                 -> refund
  otherwise                           -> deny
Record: matched_rule_id, input_row_id, log_row_id

The test is the point. A rule without a replay test is a guess with better formatting.

The mechanism is simple: take every log row, feed its input fields to the rule, compare the rule's action to the logged action. The output is a table, not a pass or fail, because you need to see which rows disagree and why.

Two disagreements out of forty is a 95 percent match, which sounds fine and is not. Row 038 shows a conflict in the log itself: a 610 dollar refund overrode the value threshold. The rule order puts value first, the human put cause first. Row 040 shows a logged escalation for a small amount on E19, where the rule refunds. Neither is a bug in the rule; both are gaps in the log or in the rule's ordering.

Set a threshold before you look at the results, and hold two of them. A match rate on seen rows tells you the rule fits history. A match rate on held-out rows (the last ten decisions, or a random fifth) tells you it generalizes. A rule that scores high on the first and low on the second has memorized the log, usually by overfitting a long chain of conditions to one unusual row.

Braintrust's guidance on testing agent systems makes the same structural point one level up: control the starting state, define a verifiable final state, and score requirements separately, so a polished output cannot hide a skipped action in their multi-agent testing guide. Apply that to a single rule by scoring the action, the matched condition and the recorded log row as three separate checks.

Log analysis is not optional at this stage. Final outcomes alone hide how the agent got there, and the paper on evaluable agent behavior argues that systematic tracking of inputs, execution and outputs is necessary because outcome-only scoring cannot tell a correct answer from a shortcut as set out in the log analysis paper. Your replay suite is log analysis applied to your own decisions.

Disagreement is the normal state, not the exception. Sort every mismatch into one of four buckets, because each has a different fix.

Bucket 1: missing condition. The human used a fact the rule does not read (a contract clause, a prior incident). Fix: add the field to the log going forward, then add the condition. Do not add the condition from one row; wait for a second independent row or a documented rule change.

Bucket 2: wrong ordering. Both conditions match, the human applied the other one. Fix: reorder and re-run the full suite. Ordering changes are high risk because they alter many rows at once, so never reorder without a full replay.

Bucket 3: log error. The recorded decision does not match the evidence attached to it. Fix: correct the log, keep the original entry marked as superseded and link to the correction, which preserves the history rather than creating two current-looking versions as described in the decision log template guidance. This bucket is why evidence links matter.

Bucket 4: genuine change. The world moved: a policy changed, a threshold was revised. Fix: version the rule, keep the old one, and add a note tying the change to its revisit trigger.

Decision logs are described as living documents that need regular updating, archiving of outdated entries and tracking of outcomes in the monday guide. A weekly review is the natural home for the drift check, and the GTD-derived weekly review structure of getting clear, getting current and getting creative maps onto triaging mismatches, updating the rule and asking whether a new condition is needed as Todoist describes the practice. A 20-minute Friday review that compares what was planned against what happened and commits to one change per week is enough to keep a rule honest per the 100tasks founder review template.

One discipline from the Eisenhower framework transfers directly: separate the decision that must be made now from the one that can be scheduled. A mismatch that would cause a wrong refund today goes in the immediate bucket; a mismatch on a rare error code goes in the scheduled bucket with a revisit trigger as the Eisenhower matrix describes the four quadrants. Without that split, every mismatch looks equally urgent and the rule stops being maintained.

Two signals worth watching between reviews. First, the share of traffic hitting the default action. If it climbs, the rule is losing coverage. Second, the age of the oldest unreviewed mismatch. Growth in either one predicts the rule will quietly diverge from human practice.

Once a rule is written and tested, the remaining work is plumbing: running the rule on every request, logging each match, replaying the suite in CI, and routing mismatches to a queue. Teams that stop here usually discover the rule works but no one can see it working.

Two light options at this stage. First, keep the rule, the test harness and the review queue in the same repository as the rest of the operational code, so a rule change goes through the same review as a code change. Second, if the rule sits in front of several model calls (a classifier for the soft judgements, an embedder for similarity, a text model for the reply), route them through one endpoint so the decision log captures every call in one format. Quanta Inference puts text, image, video, audio and embedding models behind a single OpenAI-compatible key with prepaid US-dollar pricing, which keeps the logging surface to one place at Quanta Inference.

If you would rather have the first version built with you than assemble it from parts, describing the log, the desired rule and the review queue to a specialist is a reasonable use of Quanta.