How to Interview a Vendor's Model Validation Team

Open with the short version. If the answers are vague, the rest of the hour will not fix it.

Who signs off on the model? You want a named individual, a title, and the reporting line above them. Listen for a person in risk or compliance who does not sit under the head of product or the head of the business unit that sells the model. If the answer is "the model governance committee" with no individual name, ask who chairs it and who that chair reports to.

What evidence do you keep? You want a list of artifacts with retention periods, not a description of process. Dated review notes, test scripts and results, the issue log, the change log, and the monitoring dashboards.

What happens when you disagree with the model owners? You want a written escalation route, the name of the forum that arbitrates, and at least one example of a disagreement that went up and what happened.

A useful rule of thumb: if all three answers can be given by the same person who owns the commercial relationship, you are talking to a quality assurance function, not validation.

Nobody says no to that question. Test it with structural questions whose answers are checkable.

The budget question is the one that separates real independence from the appearance of it. A validation team whose salaries are charged back to the product it reviews has an incentive to be agreeable. The OCC handbook treats independent risk management staff as a distinct function with its own oversight role (Comptroller's Handbook), and you should expect the same shape from a vendor that claims mature governance.

Ask also about effective challenge. The regulatory concept is review by someone with no stake in the outcome (Zyphe's walkthrough of SR 11-7 for compliance AI). A validation team that only reviews documentation produced by the model owners is not challenging anything. Ask what proportion of review time is spent independently reproducing results versus reading the model owner's write-up.

Documentation standards exist so that a party unfamiliar with a model can understand how it operates, its limitations and its key assumptions (OCC 2011-12, p.21, quoted by RiskSpan). That is the bar to hold a vendor to. Ask for these five artifacts and see how fast they arrive and how complete they are.

• Model documentation covering purpose, inputs and sources, assumptions, limitations and how outputs are used.
• Validation report with the reviewer's name, the date, the tests performed and the conclusion, including any reservations.
• Independent test results such as sensitivity analysis or benchmarking against a challenger model. Where a vendor model does not permit full access to the code, guidance accepts greater reliance on sensitivity analysis and benchmarking (RiskSpan), so ask which of those they actually ran.
• Issue log with severity, owner and resolution date for each item.
• Monitoring evidence showing production performance over time against thresholds that trigger re-review.

Then do the one thing that turns an interview into an audit: pick two entries from the issue log and ask the validation lead to walk you through them, including who fixed the problem and when it was closed. A log maintained for show tends to have entries that nobody can explain in detail. A real one has stories attached.

Data lineage matters here too. If the model consumes customer data, you need to know where it came from and what transformations it passed through, because lineage records that demonstrate origin, transformation and authorization are what hold up in an exam (RiskInMind on audit-ready data lineage). Ask whether lineage for the model's inputs is documented per critical data element or only at system level.

Suppose the validation lead tells you the model was reviewed before launch and passes a quarterly performance review. That sentence sounds fine and tells you almost nothing. Break it into dates and objects.

Ask: what was the date of the last full validation? What was the date of the last limited-scope review? Which specific tests were rerun, and what were the results against the threshold? What changed in the model between those two dates? The regulatory logic on scope is that changes affecting inputs or calculations generally need deeper review than changes to how outputs are presented (RiskSpan on determining validation scope). So a vendor that touched model weights six weeks ago and ran only a presentation-layer check has a gap, and you should ask them to name it.

Here is a scoring approach you can apply in the room. Give each of the six dimensions below a score of zero, one or two, and treat a total under eight as a signal that you need more of your own validation work before signing.

That scoring matters commercially because of what you are buying. Contact center AI is where banks see their biggest exposure, and customers who hit repeated bad experiences take action: in a Deloitte survey of banking customers, 28 percent said they reduced spending with the bank after negative contact center experiences and 31 percent stopped doing business with the institution altogether (Deloitte Insights). A weakly validated model in that channel is a revenue risk, not just a compliance one.

Ask for the escalation path in writing and then ask three follow-up questions.

Who sits on the committee that arbitrates? You want at least one member who does not report to the model owner. What is the service level for a decision? A path that takes a quarter to resolve is a path that discourages raising issues. Has the committee ever upheld the validation team against the model owner? If never, either the validators are never right or the process is decorative.

Also ask what happens to a model already in production when validation finds a problem. There is a difference between a team that can stop a launch and a team that can force a rollback or a restriction. The second is the stronger signal, because it means the validation function has authority over something already earning revenue.

Two more areas deserve specific questions.

Monitoring and override rates. If the vendor's customers routinely override the model's output, that is a finding in itself. Override frequency is one of the signals that should drive how often a model is revalidated (RiskSpan). Ask the validation lead for the override rate across their customer base and what threshold triggers re-review.

Performance monitoring evidence. In AML and transaction monitoring, alert volume is dominated by false positives, with industry rates routinely above ninety percent (Adjapong's AI-TMS paper on SSRN) and some studies putting it as high as 95 percent (Flagright). If the product is in this category, ask how the validation team measures alert precision in production and when they last changed a threshold as a result. A team that cannot quote a precision figure has not been doing outcomes analysis.

If you would rather spend the meeting talking to people who build and review models for a living than assemble this evidence yourself, a free consulting call can map the questions to your own model inventory before you go back to the vendor.

The first meeting should be with the validation lead alone. The second should include the model owner, because disagreements are more informative when both parties are in the room.

Watch for who speaks first. If the model owner answers questions about validation findings, the validation team has lost the floor. Ask the validation lead directly what they would change about the model if they owned it, and see whether the model owner interrupts.

Ask the two of them to describe their last real disagreement. You want a specific model, a specific issue, a date, and a resolution. Vague references to "healthy tension" mean no disagreement has been escalated.

Finally, ask what the validation team is not allowed to see. Every vendor has boundaries around source code and training data. The useful answer names the boundary and describes the compensating test, such as benchmarking output against an independent dataset. The unhelpful answer is that there are no boundaries.

Run this process before you sign, and keep the artifacts you collect. They become the vendor section of your own validation file, and they will save you weeks when the model inventory is next reviewed.