How to Review a Vendor's Model Documentation for Missing Evidence

A worked example is the cheapest form of proof and the one most often absent. "The model reduces false positives" is a claim; "here is an alert that the old rules flagged and this model scored, with the features that drove the score and the disposition" is evidence. For compliance AI, the point of validation is to let you defend a specific KYC or AML decision to a regulator rather than trust the vendor, which is why per-decision audit trails belong in the audit pack per Zyphe's validation checklist.

Ask for a worked example on a real input you can relate to, and check that it shows the input, the output and the reasoning between them. A model card's quantitative analyses section is supposed to do exactly this: report error rates separately by group and condition so a single headline score cannot hide failures concentrated in one subgroup as Ultralytics explains. If the card gives you one aggregate accuracy number, you have not been given a worked example.

Here is a minimal request you can paste into a vendor email:

Please provide, for each metric in your evaluation report:
1. One real (or faithfully synthetic) input record used in the test
2. The model output for that record, including the score and threshold
3. The expected label and the disposition that followed
4. A pointer to the exact row in the evaluation dataset

If the vendor cannot produce one traceable record, treat the metric as a marketing statement. That is a defensible position in a validation file, and it is far cheaper to assert now than to discover during an exam.

A threshold turns a claim into a test. "Well calibrated" means nothing until the vendor states the calibration metric, the value it must stay above, and the sample it was measured on. "Low false positive rate" needs the base rate it was measured against, because a rate quoted without the underlying alert volume cannot be compared to your own book.

The false positive problem in transaction monitoring is the best illustration of why thresholds matter. Industry false positive rates routinely exceed ninety percent in static, threshold-based rule systems, and the operational consequence is that investigator capacity is consumed by alert disposition rather than actual financial-crime intelligence per this SSRN paper on AI-driven monitoring. A vendor claiming a large reduction in false positives against that baseline must state the baseline, the metric and the test set, or the number is unusable.

Governance guidance makes the same demand. Ongoing monitoring is one of the three core elements of model validation, alongside conceptual soundness and outcomes analysis per the Comptroller's Handbook summary, and monitoring needs thresholds to monitor against. A vendor that supplies no numeric pass or fail line has, in effect, made ongoing monitoring impossible without you inventing the standard yourself.

When you do set your own threshold, state it in the validation file and hold the vendor to it. A workable pattern is to agree on a tolerance band, define the metric, and require the vendor to report performance against it on a fixed cadence. That converts a vague assurance into a testable obligation.

Every section of the documentation needs a human behind it. Not a department name, not a support inbox, a role or an individual who can answer a challenge and whose name appears in your validation file. Model risk guidance is explicit that accountable ownership is part of what validation expects, alongside documentation of limitations and monitoring per Zyphe's summary of SR 11-7.

A named owner is what makes the change log usable. When a model changes, the first question is what about the model changed, and whether that change affects inputs or calculations, which is more likely to require a fuller review than a change that only affects how outputs are interpreted as RiskSpan sets out. Without an owner, you cannot ask that question of anyone in particular, and changes drift in unannounced.

The practical habit is to open the documentation and, next to each section heading, write the name you were given. Sections that produce a blank are your first round of questions. In most vendor packs you will find the model details section names a contact, while the training data, evaluation and limitations sections name nobody. That pattern alone tells you where the vendor expects scrutiny and where it does not.

The documents arrive in a predictable shape, so you can review them in a predictable order. The table below maps each document to the sections where claims usually lack a worked example, a threshold or an owner, and what to ask for.

Model cards are a good starting point because the format is standardized even when the content is thin. The original format proposed nine sections, including intended use, factors, metrics, evaluation data, training data and disaggregated quantitative analyses as Ultralytics details, and governance teams reasonably expect those sections to be present. Presence is not the same as evidence, which is why you read each section against the three tests rather than ticking that it exists.

Data provenance deserves particular attention because it is where vendors are most tempted to go shallow. Lineage only holds up when it traces a data element from origin through every transformation to its use in a report or decision, with control metadata showing who touched the data and under what authority per RiskInMind's lineage roadmap. A diagram of systems is not lineage; a field trace with named stewards is.

Evaluation reports are the most technical document in the pack and the easiest to over-trust. Four traps recur.

First, the test set. A metric measured on public or synthetic data may say nothing about your population. Ask how the test data differs from your own book and get a concrete answer. Second, the baseline. A claimed improvement needs a stated comparator, because a number quoted without the baseline cannot be benchmarked. Third, the operating point. A detection rate is meaningless without the threshold it was measured at, since you can always trade one for the other. Fourth, disaggregation. Aggregate numbers can hide a subgroup where the model is unusable, which is the whole reason model cards call for quantitative analyses broken down by factor.

A short checklist keeps the review honest:

• Does each headline metric name its test set, sample size and measurement date?
• Is there a baseline and is it the same rule or model in production today?
• Is the operating threshold stated, and can a reviewer move it and see the effect?
• Are results disaggregated by segment, and does the worst segment pass your threshold?
• Is every number tied to a named owner who can explain it?

You can strengthen the review by asking for sensitivity analysis: how the model behaves as a key input varies. Testing methods for effective challenge on vendor models often rely on sensitivity analysis and benchmarking precisely because full access to the code is not available as RiskSpan notes. If the vendor will not share code, sensitivity testing is the substitute, and a report that omits it leaves you with nothing to challenge.

The output of your review is not a verdict, it is a list of findings written so that a colleague can act on them. For each gap, record the claim as stated, the test it failed, the evidence you requested and the date you asked. That structure survives an exam because it shows a control operated, not just that you had an opinion.

Effective challenge is the standard you are trying to meet. Validation performed simply to tick a box tends to settle for the cheapest option, while validation driven by risk and complexity asks harder questions about data, assumptions and whether the model keeps performing over time as Darling Consulting argues. Your follow-up emails are where that difference shows, so write them as specific requests rather than general concerns.

A few follow-up questions that surface gaps fast:

• Which single input, if it changed, would move this metric the most, and what is the measured effect?
• If I gave you an alert from my own queue, would you show me the score and the features behind it?
• Who signs off when this model changes, and what triggers a full versus a limited review?
• What is the worst-performing segment, and what does the model do when it meets it?

For AI-specific use cases, add one more: how does the system explain a single decision to a reviewer or a regulator? Explainability is now part of what makes alerts trustworthy, and the trend in the market is toward models that explain why an alert fired rather than only that it did as Facctum notes. A vendor that cannot walk through one decision has not given you the audit trail you need.

Here is how the tests play out on a realistic submission. A vendor sends a model card, a six-page evaluation report, a two-page data provenance note and a change log with quarterly entries. You review in an afternoon and can see immediately where the pack is thin.

The model card's intended use section lists out-of-scope uses without consequence, so you ask for one worked example of an out-of-scope use and what happens. The evaluation section shows a single aggregate detection rate, so you ask for disaggregation by customer segment and condition, with the threshold at which it was measured. The limitations section says "performance may vary with data quality" with no measured boundary, so you ask for the condition where the drop becomes material and the number. The data provenance note maps systems but not fields, so you ask for a field-level trace for the three data elements that feed the alert score, with named stewards.

On the change log, the entries say "improvements to scoring" and "data refresh." You ask which of those changes affected calculations versus reporting, since that distinction drives whether your next review is full-scope or limited per RiskSpan's four questions. You also ask who signed off each entry, which is the owner test in practice.

Now the arithmetic that makes the finding concrete. Suppose the vendor claims a forty percent reduction in false positives and you currently generate 1,000 alerts a month. If the baseline is not stated, you cannot convert forty percent into anything. If the vendor states it measured against a rule-based baseline with a ninety percent false positive rate, as industry rates commonly run per this SSRN paper, the claim becomes checkable: you know what was measured, on what, against what. Without those three numbers the claim is unproven, and you write it up that way rather than passing it through.

The findings output is short and specific:

That table is what you attach to the validation report. It is defensible, it is reproducible, and it gives the vendor a clear list to answer.

If you are building the machinery behind this kind of review, for example running your own challenger evaluations across several vendor models, the plumbing of calling many different models through one interface matters more than it looks. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key (https://quantaintelligence.ai/inference), which is convenient for standing up a challenger benchmark without a separate integration per model. The evaluator still has to be you. The API just removes the wiring.