A Regression Set for Table-Driven Inference

You cannot test every combination, and you should not try. A decision-table approach to deriving test cases from requirements exists precisely because redundant cases can be trimmed substantially while coverage is kept; one study reports redundancy reductions of up to about a third when cases are derived from a decision table rather than invented ad hoc the decision-table reduction paper. Use that as your selection filter: each case must be able to fail for a distinct reason.

Apply this rule to selection. Add a case only if you can state, in one sentence, the specific edit that would make it fail.

Use four sources, in this order.

Boundaries. Take every numeric threshold in the table and test at the threshold, one below, and one above. If a rule fires at 30 days, you need cases at 29, 30 and 31. Boundary value analysis is a standard black-box technique for exactly this reason the Go wiki does not cover thresholds, but the boundary technique is named alongside decision tables in the testing literature the decision-table reduction paper.

Overlaps. Find inputs that satisfy two rules at once and record which one wins. Overlap resolution is where table edits most often cause silent behavior change, because reordering rows changes the answer without changing any cell value.

Previously wrong cells. Every production incident gets a permanent case. This is the highest-value category and the one most teams skip.

One boring case per rule. If a row never appears in any passing case, you have no evidence it works at all.

A comparison of the four sources:

The failure mode that kills these sets is not a missing runner. It is the habit of regenerating expected outputs from the current table whenever the check goes red. That converts the test into a rubber stamp.

Three practices prevent it.

Version the set with the table, in the same commit or next to it. Structured-prompt-driven development treats prompts as first-class artifacts kept with the code in version control, reviewed like any other change Martin Fowler on SPDD. The same discipline applies to a rules table and its cases: if the table and the set can diverge in history, nobody can tell which one is authoritative.

Make expected-output edits a separate, reviewed change. A diff that touches both a rule cell and the case asserting that rule is a red flag. Require the author to name the rule change in the pull request. This mirrors the iterative review skill that the SPDD authors identify as necessary for governable LLM-assisted change Martin Fowler on SPDD.

Freeze the output shape. If the shape is unstable, every field rename breaks every case and people start regenerating. Schema-based extraction guidance makes the point that a schema is a contract you define once and reuse, with field names, types and descriptions stated explicitly schema-based document extraction. Define your output contract the same way, and reject edits that add fields to the output without a case covering them.

For comparison, normalize before you diff: sort keys, trim whitespace, and cast numbers to one type. Everything else should be exact. If a field genuinely has a set of allowed values, use an enum in the contract rather than a fuzzy match in the runner, which is the same advice given for schema design in extraction systems schema-based document extraction.

The runner has four steps and no cleverness: load the table, load the cases, evaluate each input through the same inference path production uses, compare, report. Never reimplement rule evaluation in the test harness. If the runner and production read the table through different code, you are testing the wrong thing.

import json, sys

def run(cases, infer):
    failures = []
    for case in cases:
        got = infer(case["input"])
        if got != case["expect"]:
            failures.append({
                "name": case["name"],
                "input": case["input"],
                "want": case["expect"],
                "got": got,
            })
    return failures

if __name__ == "__main__":
    cases = load_cases("cases/rules.yaml")
    failures = run(cases, infer_from_table("rules/table.csv"))
    print(json.dumps(failures, indent=2))
    sys.exit(1 if failures else 0)

Two details raise the runner's value. First, use subtests or named cases so the output tells you which case failed without reading the harness, which is the reason the Go example passes a name into each subtest the Go wiki. Second, report input, wanted and got together. The Go authors call this out explicitly: a careful test with good error messages is what makes the amortized cost worth it the Go wiki.

Keep the set fast. It should run in well under a minute. If it does not, you have either too many cases or a network call inside the runner. Regression suites are meant to run whenever a change is made, which is only practical with automation Leapwork's regression testing guide.

A check nobody runs is decoration. Two insertion points cover the realistic cases.

The edit path. When an editor saves the table, run the set against the proposed table before committing. Show failing cases with their names and diffs. This is confirmation plus regression in one step: it tells the editor whether their intended change worked and whether anything else moved. The distinction matters, because regression testing is aimed at consequences you did not intend, not at the change itself PractiTest on regression testing.

CI on every change to the table or the cases. Fail the build, not the deploy. Gate the merge. The same source notes that each change to a tested artifact invalidates previous test results, which is why the suite runs on every change rather than on a schedule PractiTest on regression testing.

If inference calls a model, put the model call behind a fixed, versioned prompt and pin the model for the test run. Prompts kept as first-class artifacts under version control make this possible Martin Fowler on SPDD. Where the model's output is a JSON object, a schema-first contract reduces variance in the shape and lets you compare exact objects schema-based document extraction. Systems that optimize the schema itself for machine consumption show why this matters: ambiguous, human-oriented schemas produce inconsistent output, and refining the contract improved extraction accuracy substantially in one Amazon study PARSE on schema optimization.

If you run several models behind one interface, testing against each is simpler with a single API surface. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible, prepaid key priced in US dollars, which keeps the runner's client code constant across models: https://quantaintelligence.ai/inference.

Some failures are real and expected: you deliberately changed a rule, and the case asserting the old behavior should now fail. Plan for this instead of discovering it at 6 p.m.

Use three labels on every failing case.

Two rules keep this honest. First, an intended-change update must say which rule changed and why, in the commit message. Second, if more than a small fraction of the set fails on a single edit, stop and inspect the table rather than the cases. A wide blast radius usually means a shared threshold or a reordered block, and that is exactly the moment to slow down.

There is a second reason not to skip this: regression testing exists because you cannot fully predict the impact of a change. PractiTest's framing is that since perfect impact analysis is impossible except in trivial cases, you test unchanged functionality to catch unintended consequences PractiTest on regression testing. A table edit looks local and is not.

Finally, prune deliberately. If two cases fail for the same edit every time, one is redundant. Use the decision-table view of your rules to spot duplicate coverage rather than deleting cases because they feel old the decision-table reduction paper.

You should be able to answer three questions in under a minute: which cases cover this row, what changed the last time this case failed, and who approved the last expected-output edit. If you can, the set is doing its job. If you cannot, the runner is fine and the curation is not.

For teams where the rules table spans several domains and several editors, the review burden grows past what one engineer can hold. A specialist who reads your context and returns finished work, including a curated case set and a wired CI check, is one option: https://quantaintelligence.ai/agents.