How to Test a Model's Claimed Performance on Your Own Data

The population you sample from shapes the answer more than any other choice. Pull from the decisions your team actually made, in the period you care about, including the boring middle of the distribution. Three properties matter:

• Eligibility. Every row must have been eligible for the vendor's model at decision time. If your process changed (a new document requirement, a new product line), split the population at the change rather than mixing.
• Label availability. You need ground truth. If labels arrive late (chargebacks, defaults, audit outcomes), sample from a cohort old enough that outcomes have matured.
• Randomness. Sample with a documented random seed and record it. A convenience sample of the last two weeks of tickets will not reproduce.

A defensible default is a simple random sample of 400 to 1,000 decisions, stratified so that the rare class (fraud, exceptions, escalations) is not starved. Use stratification only for sampling, then reweight when computing the metric, or your numbers will not match the vendor's population. Keep a written record of every filter you applied. Data lineage work in regulated settings shows why: examiners want origin, every transformation, and who authorized each change, not just a final number the lineage guide for banks.

A worked sizing example: if you expect precision near 0.90 and want a 95 percent interval roughly 3 percentage points wide, you need about 380 positive predictions, not 380 rows. At a 20 percent positive rate, that is around 1,900 decisions. This is the step most teams skip, and it is why their first results look alarming.

Split the sample into two parts before anything else happens:

Run the calibration slice first. If your recomputation of the vendor's own worked example does not land within a couple of points of their stated result, your pipeline is wrong somewhere: the threshold, the label mapping, or the preprocessing. Fix that before you touch the holdout. The whole point of a holdout is that it stays clean, and a holdout you have already tuned against is no longer a holdout why the data used to evaluate a model matters.

Two mechanical rules that prevent the most common failures:

• Never let the vendor see holdout predictions or labels before you have written your number down. Send hashes if you need proof of what you evaluated.
• Never adjust anything after scoring the holdout. If you change the threshold, you have created a new experiment and need a new holdout.

For text and document models, hold out whole documents rather than passages, and preferably whole counterparties or accounts. Chunk-level splits leak because passages from the same document share phrasing. Evaluation sets only measure generalization when they are strictly separated from anything used in development LlamaIndex on evaluation datasets.

Labeling deserves its own pass. Have two people label 50 holdout rows independently and measure agreement. If they disagree on more than roughly one row in ten, your label definition is ambiguous and the vendor's number is not comparable to yours until you resolve it. This is exactly the failure mode described in model card practice: a single headline score hides failures concentrated in one subgroup, which is why disaggregated evaluation is the core of the idea Ultralytics on model cards.

Now reproduce the metric. Suppose the vendor reports precision at a fixed threshold: of the cases the model flags, what share are true positives. That definition is unforgiving. It depends on the threshold, the positive class, and which rows were eligible. Change any one and you are measuring something else.

import numpy as np
from sklearn.metrics import precision_score, recall_score

# y_true: your adjudicated labels (1 = should have been flagged)
# y_score: vendor scores on the SAME scale and threshold they published
THRESHOLD = 0.70  # take from vendor documentation, not intuition

y_pred = (y_score >= THRESHOLD).astype(int)
precision = precision_score(y_true, y_pred, zero_division=0)
recall = recall_score(y_true, y_pred, zero_division=0)

n_pos_pred = int(y_pred.sum())
# Wilson interval for precision
z = 1.959964
p = precision
denom = 1 + z**2 / n_pos_pred
centre = (p + z**2 / (2 * n_pos_pred)) / denom
half = z * np.sqrt(p * (1 - p) / n_pos_pred + z**2 / (4 * n_pos_pred**2)) / denom
print(f"precision={p:.3f} n_flagged={n_pos_pred} CI=({centre-half:.3f}, {centre+half:.3f})")

Two things in that snippet matter more than the code. First, the interval is computed on the number of flagged cases, not the number of rows. Second, if the vendor compared against a threshold you cannot see, ask for the score distribution on their example and infer it. Do not silently pick a threshold that flatters the model.

Accuracy alone misleads on imbalanced data, and a single score can hide important failure modes, which is why precision, recall, and F1 are tracked together for classification and paired with ranking metrics when ordering matters Label Studio's evaluation guide. Recompute the vendor's headline metric first, then add two or three secondary metrics of your own so you can see whether a matching headline is hiding a trade you would not accept.

Then break the result down. Compute precision by product, by region, by document type, and by language. Vendor models frequently import assumptions that do not reflect the behavior of a specific customer base, and a validator that only runs aggregate tests will miss it Darling Consulting on validator questions. If your aggregate precision is 0.88 and one product sits at 0.61, the aggregate number is technically true and practically useless.

Put the two numbers side by side with their conditions, and do the arithmetic before you have an opinion.

If the gap is smaller than the interval width, your data has not contradicted the vendor. If it is larger, you have a real question, and the next move is diagnosis rather than rejection: check the population mix, check label drift, and check whether the vendor's demo excluded a category your population is full of. Structural differences are the usual culprit, and they are fixable with configuration or retraining.

Track the cost side too. A precision gap of a few points translates into a specific number of extra false positives per thousand decisions, and each one costs reviewer time. Convert the metric gap into that number before the commercial conversation, because it reframes the debate from "is the model good" to "what does the difference cost per month."

One more control: rerun the same holdout on a second model, ideally a different provider, using an interface that returns comparable scores. Benchmarks help, but they diverge from production reality, and a challenger run on your own labels is the strongest single piece of evidence you can put in front of a committee Voxel51 on evaluation practice. When you need to compare several providers without rebuilding every integration, running them through one OpenAI-compatible endpoint such as Quanta Inference keeps the scoring pipeline identical and the comparison honest.

The result is worth little if you cannot defend it in six months. Assemble a small evidence pack:

• The sampling script and seed, plus the population definition and date range.
• The label rubric, inter-rater agreement figures, and who adjudicated disagreements.
• The vendor's metric definition as received, with the date and the person who supplied it.
• Scores and labels for the holdout, stored immutably.
• The recomputation notebook, so anyone can rerun it.
• A one-page memo stating your number, the interval, and the decision you recommend.

This is close to what regulated firms already produce. Comprehensive validation covers conceptual soundness, ongoing monitoring and benchmarking, and outcomes analysis, and anything short of all three is a limited-scope review RiskSpan on validation scope. Make clear in your memo which of the three you actually did, because claiming more than you performed is worse than scoping honestly.

Then put the evaluation on a calendar. Model performance decays as behavior shifts, and ongoing monitoring with periodic re-evaluation on fresh samples is the only way to catch it before it costs you Voxel51 on continuous evaluation. A quarterly rerun of the same pipeline, with a frozen regression subset, is enough for most teams.

Once the numbers are in, resist the urge to negotiate on vibes. Use a rule like this:

• Gap inside the interval: accept the vendor's claim as consistent with your data. Move on to cost and integration.
• Gap larger than the interval, explained by population mix: ask the vendor to reprocess a slice configured for your population, then re-evaluate on fresh rows.
• Gap larger than the interval, unexplained: escalate. Request the disaggregated results and the exclusion rules. A vendor who cannot produce them has a documentation problem that regulators will find later ValidMind on model risk management.
• Any gap where one of your subgroups collapses: treat as a failure regardless of the aggregate. Aggregate wins do not offset a segment at half performance.

If you want a second opinion on the design before you commit reviewers to labeling, a short working session is cheaper than a botched evaluation; a free consulting call can cover sampling, label rules, and the metric definition in one sitting.