How to Build a Reusable Harness for Vendor Model Tests

A fixture, in software testing, is the fixed environment and known data a test runs against so that results are repeatable (Stack Overflow). For model testing, the fixture is a directory of cases: an input, the expected or reference output, and metadata that tells the scorer how to judge it.

Keep fixtures in version control with the harness code. Each case is one file or one row, with a stable identifier. When you change a fixture, the change is visible in the diff, which is the point: silently editing a test case after seeing a model fail is the model-testing equivalent of moving the goalposts.

Design principles that hold up:

• Broad coverage beats volume. A few hundred well-chosen cases across the failure modes that matter (edge cases, ambiguous inputs, long inputs, inputs in other languages or formats) tell you more than thousands of near-duplicates.
• Disaggregate by slice. Model evaluation guidance recommends measuring performance across meaningful slices such as regions, languages, device types, or content categories so you can see where performance breaks down (Label Studio). Store the slice label on every fixture row.
• Separate the stable regression set from the exploratory set. Keep a frozen subset that never changes so runs stay comparable, and a larger evolving set for diagnosis (Label Studio).
• Record provenance. Each case should note where it came from and whether it is synthetic or real, so a reviewer can weight it appropriately.

Here is a workable fixture layout:

harness/
  fixtures/
    cases.jsonl          # one case per line, stable IDs
    regression_ids.txt    # the frozen subset
  run.py                 # calls the model, writes raw outputs
  score.py               # reads raw outputs, applies metrics
  report.py              # builds the comparison table
  runs/
    2024-11-vendor-a/
      raw.jsonl
      scores.json
      report.md

Each line in cases.jsonl carries an id, the input payload, the expected output or a rubric reference, the slice label, and a provenance note. Keep the input payload as data, not as a template string buried in code, so you can diff two versions of the same case later.

The single most important architectural decision is to split the harness into two stages. Stage one calls the model and writes raw outputs to disk. Stage two reads those outputs and applies metrics. They never run in the same process, and they never share state.

Why this matters: if scoring is entangled with the model call, any change to the scoring code changes the meaning of every past run. If the model call fails midway, you cannot tell whether the numbers you got are real or artifacts of a retry. If you want to re-score an old run with a new metric, you cannot, because the outputs were never persisted in a form you can reprocess.

Persisting raw outputs also lets you audit disagreements. When two runs produce different scores, you open the raw files, find the case IDs where they diverge, and read the actual outputs side by side. That is the difference between "vendor B scored higher" and "vendor B handled 31 of 34 ambiguous date inputs correctly while vendor A fell back to a default."

A minimal two-stage harness in Python looks like this:

# run.py: call the model, persist raw outputs, do nothing else
import json, hashlib

def run(model_call, cases, out_path):
    with open(out_path, "w") as f:
        for case in cases:
            out = model_call(case["input"])          # the only model-dependent line
            f.write(json.dumps({
                "id": case["id"],
                "slice": case["slice"],
                "input_hash": hashlib.sha256(
                    case["input"].encode()).hexdigest()[:12],
                "output": out,
            }) + "\n")

# score.py: read raw outputs, apply metrics, know nothing about the model
import json

def score(raw_path, fixtures, out_path):
    expected = {c["id"]: c for c in fixtures}
    results = []
    for line in open(raw_path):
        row = json.loads(line)
        case = expected[row["id"]]
        results.append({
            "id": row["id"],
            "slice": row["slice"],
            "pass": judge(row["output"], case["expected"], case["rubric"]),
        })
    with open(out_path, "w") as f:
        json.dump(results, f)

The judge function is where you put exact match, structured comparison, or a rubric-based check. Keep it deterministic where you can. If you must use an LLM as a judge, record the judge model and prompt alongside the scores, because a change to the judge changes every historical score.

Comparability is fragile. A harness that produces a score you cannot compare to last quarter's score is only slightly better than a one-off test. Protect comparability with four habits.

First, freeze the fixture set per run. Record a hash of cases.jsonl in every scores file. If the hash changed between two runs, the comparison is qualified, and the report should say so.

Second, control for data leakage. Evaluation data must be kept strictly separate from anything the model was trained or tuned on, because overlap invalidates the results (LlamaIndex). With vendor models, you usually cannot verify training data, so the practical defense is to keep some fixtures private and never share them with the vendor, even during setup. If a vendor suddenly aces a case it failed before a release, check whether that case was visible to them.

Third, pin the run conditions. Record the date, the endpoint or version identifier the vendor exposes, sampling settings, and any system prompt used. A vendor that changes a default temperature between releases has changed your test conditions, not just their model.

Fourth, do not tune against your final set. In classical evaluation, once a test set has been used to adjust a model, its metrics are no longer unbiased and it can no longer serve as a true test (Alteryx Community). The harness version of this is: if you revise prompts or post-processing to fix a failure you saw in the frozen regression set, move that case into the exploratory set and accept that the frozen set has been compromised for future comparisons.

A useful maintenance cadence:

That last row is where a harness earns its keep. A vendor that silently changes behavior between releases will show up as a score movement on a set of cases nobody touched.

A report is not a dump of numbers. It answers three questions: which model won on which slices, where the gap is largest, and whether the difference is bigger than the noise in your test set. Structure the report with a headline table, then per-slice tables, then a short list of the cases where models diverged most sharply.

The headline table should show every run you want to compare, with the metric version and fixture hash in the header so nobody compares two incompatible numbers by accident.

The numbers above are illustrative, not benchmark results. The point is the shape: same fixture set, same metric version, side by side, with the weakest slice surfaced because that is where a decision usually turns.

Two report conventions make the output usable by people who were not there when the test ran. Always show the denominator, because a 61% pass rate on 18 cases is a very different claim from 61% on 180. And always show the divergence list, sorted by how many runs got a case right versus wrong, because the cases where all vendors fail tell you your fixtures are testing something hard, and the cases where exactly one vendor succeeds are where the real signal lives.

The tempting shortcut is to reuse a public benchmark. Public benchmarks are useful for orientation, but they can diverge from production reality because domain shift, adversarial inputs, and new user behaviors create failure modes a curated test set never captures (Label Studio). The same talk on out-of-distribution evaluation makes the point formally: a held-out test set only gives you a rigorous guarantee if the new data really comes from the same distribution as the old data (ComputerVisionFoundation).

A second shortcut is to trust the vendor's model card. Model cards are genuinely useful: they document intended use, evaluation data, disaggregated performance, and known caveats, and the original proposal defined nine standard sections including quantitative analyses broken down by relevant factors (Ultralytics). The IAPP notes that model cards have no mandatory or established requirements, which is exactly why they are an input to your evaluation rather than a substitute for it (IAPP). The Snowflake guide makes the same distinction: a model card records where a model was evaluated, on which populations, and where it is likely to underperform, and it is a versioned artifact tied to a specific model version (Snowflake). Use it to choose which slices your fixtures must cover, then measure those slices yourself.

A third shortcut is single-score reporting. A single headline score hides failures concentrated in one subgroup, which is the argument for disaggregated evaluation in the first place (Ultralytics). Your report should never show one number without at least the per-slice breakdown underneath it.

Run the harness on four occasions: before a purchase decision, when a vendor announces a change to a model you depend on, when a failure in production suggests a gap in your fixtures, and on a fixed calendar cadence so silent drift gets caught. Each run produces a report filed next to the raw outputs, and the reports accumulate into a history that answers the question nobody asks at purchase time but everyone asks later: is this model still as good as it was when we chose it.

The operational details that keep a harness from decaying:

• One command per stage. If running the harness takes a wiki page of instructions, it will not run. run.py, score.py, and report.py should each be a single command with defaults that point at the right directories.
• Store raw outputs, always. They are the only artifact you cannot regenerate if a vendor retires a model version.
• Version the metrics. A metric change is a release. Label it, and re-score prior raw outputs so the history stays coherent.
• Keep a changelog next to the fixtures. Every added case gets a line: what failure mode it covers and why.
• Assign an owner. A harness with no owner becomes a directory of stale JSON within two quarters.

If your test set spans several modalities (text prompts, image inputs, audio clips), the collection stage is where vendor APIs differ most, and that is the step a unified API removes. Quanta Inference exposes text, image, video, audio, and embedding models through one OpenAI-compatible key priced in US dollars, which means the collection script stays the same when you swap vendors and the only thing that changes is the model identifier.

The bigger organizational step is deciding what to do with the reports. A harness that produces a comparison table and no decision rule is still a research project. Write the rule down before the first run: for example, a vendor must beat the incumbent on the frozen set overall and not fall below it on any single high-priority slice. That rule is what turns the harness from a tool for testing vendors into a tool for defending the choice.