How to Turn a Replay Result into a Ship or Hold Argument

Before the memo can argue anything, the reader needs to know what the replay actually computed. Logged decisions come with a treatment assignment mechanism, and a replay that ignores it is answering a policy-evaluation question under an untested assumption. State the estimand in one line: "the change in decision outcomes if the candidate model replaced the champion on the same population, holding all downstream processes fixed."

Then list the assumptions that line depends on, ranked by how much a reviewer would care. Typical entries are that the logged population resembles the launch population, that the policy did not change customer behavior in ways the log does not capture, and that no interference exists across cases (for example, that approving more applicants does not change the applicant pool itself). If your replay used propensity weighting or a doubly robust estimator, say so and say what happened when you checked support. If it did not, say that plainly and let the hold recommendation follow from the gap.

The same study is explicit that favorable offline evidence and actionable evidence are different things, and that a support-aware framework should quantify favorable and unfavorable evidence separately before classifying launch readiness (arXiv). Mirror that structure in the memo. A section headed "what the replay supports" and a section headed "what the replay does not support" is more persuasive than one blended narrative, because it shows the reviewer that you have already looked for the disconfirming evidence.

Versioning discipline matters here too. Prompt and policy artifacts should be immutable and carry a unique identifier, so a reader can tell which exact configuration produced the replay (prompt versioning practice). The memo should cite the candidate artifact by its version identifier, not by a nickname that will drift.

The memo has to fit one page, which means every section earns its space or gets cut. The following skeleton works for credit, fraud, pricing, routing and eligibility decisions alike.

Keep the recommendation section to a single sentence plus the conditions under which it holds. A memo that says "ship, provided the review queue stays under 150 cases per day and the top two reason codes remain stable for 30 days" gives the approver something to sign. A memo that says "ship, subject to continued monitoring" gives them nothing.

The decision-log practice from engineering teams generalizes well: an entry records the decision, the owner, the date and the trigger that would reopen it, and it is kept deliberately short so a new reader can absorb months of context in ninety seconds (decision log template). A replay memo is a decision log entry with a worked evidence section attached. Formatting it as an architecture decision record, with numbered title, status and consequences, makes it storable and searchable alongside other decisions (Microsoft's decision log guidance).

For any decision that produces an adverse action notice, the reason code table is not a reporting nicety. It is the compliance surface. Regulation B requires a statement of specific reasons, and the CFPB has stated that model complexity does not excuse the requirement (the CFPB circular coverage). A candidate model that shifts the distribution of disclosed reasons changes what thousands of applicants are told about their own creditworthiness, and the memo must show that shift before approval, not after.

Build the table with one row per reason code and four numeric columns: share of declines in the champion, share in the candidate, percentage point change, and the candidate's top-decile concentration in that code. The last column catches codes that are rare overall but dominate a specific segment, which is where disparate impact arguments begin.

A worked example, with illustrative numbers you should replace with your own replay output:

Read this table as an argument, not a report. A seventeen point increase in a cash-flow reason code means the candidate model is telling a different story to a large group of applicants, and that story had better be accurate for each individual case. The Trussed guidance is blunt on this: the mapping from model attributions such as SHAP values to applicant-facing reason language must be governed, version controlled and validated, and Appendix C's sample codes are a starting vocabulary rather than a compliance ceiling (specific adverse action reasons for AI lending models).

Three checks belong under the table, each one sentence. First, whether every moved code still maps to a validated plain-language phrase. Second, whether any code's movement concentrates in a protected or monitored segment beyond an agreed threshold. Third, whether the candidate model produces reason codes for cases where the champion produced none, which happens when a model declines on a factor the champion never evaluated. If any check fails, the recommendation flips to hold or redesign regardless of the headline metric, and the memo should say so in the recommendation line itself.

Every replay leaves a residue. Some case types are too rare in the logged window to estimate, some depend on information the log did not capture, and some involve judgment that no policy should automate. The memo's credibility rests on naming them.

Use a short list with three fields per item: case type, estimated monthly volume, and named review owner. For example: applications with no bank-feed connection but two or more tradelines, roughly 400 per month, owned by the underwriting operations lead. Or: disputes where the merchant category is travel and the amount is under the chargeback threshold, roughly 120 per month, owned by the fraud operations manager. Volumes come from your own log, and the memo should state the window they were counted over.

The operational question underneath the list is capacity. If the candidate model routes 400 new cases per month into a queue that currently handles 250, the ship recommendation is really a staffing recommendation in disguise. Say that. A reviewer who approves the model without approving the queue will discover the gap in production, and the memo that hid the gap will be the one people remember.

Decision logging practice offers a useful pattern here: record the decision boundary, meaning the point at which the system crosses from interpretation into a consequential branch or side effect, and store a durable record of the planned action before the side effect occurs, then append the verified outcome afterward (AI agent decision logs). The same two-phase shape works for replay review cases: record the referral reason and the reviewer's disposition, not just a final label.

For the reviewer-facing side, hold queues should show the case summary, the evidence references, the policy reason codes and the expected change, with an expiry so stale referrals do not accumulate (same source). The memo can then state the review burden as a number of minutes per case times cases per month, which is a figure an approver can act on.

The recommendation line should pick from a small closed set. Ship means the candidate replaces the champion on the full population. Ship behind a holdout means the candidate takes a defined share, with a pre-registered comparison. Online validation means the candidate is tested on live traffic with the champion as the reference and no irreversible customer-facing change. Redesign means the evidence is insufficient and the candidate returns to development.

The distinction between ship-behind-holdout and online validation is the one teams blur most often. Champion/challenger designs deploy the current strategy and the challengers simultaneously, assign cases at random within a segment, and track performance so that underperforming variants can be stopped early (champion/challenger experiments). That is the right shape when the decision's outcomes are observable quickly and the downside is bounded. It is the wrong shape when a bad approval cannot be recalled, in which case the honest action is to hold and gather the missing evidence first.

A decision rule you can apply directly:

1. If the reason code table passes all three checks and the unresolved case volume fits within review capacity, recommend ship or ship behind a holdout.
2. If the reason code table passes but review volume exceeds capacity, recommend online validation with a capped share, and attach the staffing request.
3. If any reason code check fails, recommend hold or redesign, regardless of the metric.
4. If the replay's estimand depends on an assumption you cannot test with logged data, recommend online validation even when everything else passes.

That last rule is where the marketplace study lands: the framework found a policy with a large replay lift and a stable out-of-time result, and still declined to recommend launch, because propensities and bidder response were unverified (arXiv). Adopting that posture costs you speed in some launches and saves you from a class of failures that replay cannot see.

The memo should also state the reopen trigger in one line, because that is what turns a decision into a decision record. Examples: the top two reason codes drift by more than five points in any rolling month; the review queue exceeds capacity for two consecutive weeks; the holdout comparison falls below the pre-registered threshold at the first read. A decision log without a reopen trigger becomes a document nobody consults, which is exactly the failure mode the project management literature warns about when teams confuse a log with meeting notes (decision log versus meeting notes). The monday.com guide makes the same point about capturing rationale, alternatives and expected outcomes so the log stays useful as people change roles (decision log best practices).

Four patterns show up repeatedly, and each has a cheap fix.

The first is the metric-only memo. It reports lift, confidence intervals and a chart, and leaves the reader to infer the action. The fix is the decision statement at the top and the closed action set at the bottom.

The second is the orphaned reason code. A code moves in the table and nobody checks whether the plain-language phrase still describes what the model did. The fix is a named owner for the mapping layer and a validation note in the memo, following the governed mapping approach described for AI lending models (Trussed). Regulation B's commentary permits a creditor to disclose a reason even where the relationship between that factor and creditworthiness is not obvious to the applicant, which gives some flexibility, but the disclosed factor must still be one the creditor actually considered and scored (CFPB on adverse action notices with AI/ML models). A code that no longer reflects a considered factor is a defect, not a wording problem.

The third is the invisible review queue. The memo says "a small number of cases will be reviewed manually" without a number. The fix is the volume-and-owner list, with the capacity comparison written out.

The fourth is version drift. The replay was run on one model artifact and the memo describes another, or the prompt template changed between the run and the write-up. Immutable version identifiers and a one-line artifact reference close this gap (prompt versioning).

For teams running several candidate models at once, the cost dimension is worth a line in the memo. A measured study of evaluation configurations found that agentic evaluation with growing context replayed roughly 3.6 times the input tokens of a single structured pass, at about 3.5 times the cost per cycle, and that majority voting over independent passes reduced a measured borderline flip rate to about 10.4 percent at three votes while still converging to a floor rather than to zero (agent token cost study). The memo implication is simple: if a decision hinges on borderline cases, spending on more voting passes buys a smaller error margin, not certainty, and the residual should be disclosed in the unresolved cases section rather than averaged away.