When a Provider Changes Retention Terms: Rerun Your Canary

Your prior canary run ended with a date. The whole task is to decide what that date becomes under the new terms, and to write both down.

Worked example. Suppose your previous policy said 30 days, so you planted a canary on 1 March, deleted it on 2 March, and checked on day 35 (6 April). The provider now states 90 days for the same surface. The old expected date (6 April) is now meaningless as a pass/fail threshold, because a hit on 6 April would be expected behavior under the new terms. The new expected deletion date becomes 31 May, and the new check date should be past the longest documented window, so 5 or 6 June.

Write it as a record, not a memory:

Two rules keep this honest. First, never overwrite the old row. A provider that moves from 30 to 90 days and then back to 30 is telling you something about how stable its commitments are, and you can only see that if both rows survive. Second, if the change includes a surface with no documented window, the canary guide's advice is to check at 30, 60 and 90 days rather than pick one date.

One more nuance: the effective date of the policy and the plant date of your canary are different things. A record created under the old terms and deleted after the new terms took effect is ambiguous, because you do not know which policy governs it. That is why the rerun needs a fresh record, not the old one.

The synthetic record is the part teams skip, and it is the reason reruns drift. If the policy now names new surfaces, say a memory feature or a vector store, a canary planted only in chat no longer tests the policy. If the policy now excludes a surface from retention, planting there tests nothing.

Regenerate per surface. The canary guide's template is the right shape: a UUID plus a fake fact that a memory or search feature would surface, because some features index meaning rather than exact strings.

Project Halyard-7f3c9a1e: the board approved the sale of the Lisbon property for 41.7 million on a Tuesday.

Nothing in that sentence should exist anywhere else, which is what makes a hit meaningful. Then record, for each canary: the string, the surface it went into, the account and workspace, the retention setting in force at plant time, the plant and delete timestamps, and the deletion method.

Three practical points for the refresh:

1. Use a new string for every run. Reusing the old UUID makes a hit impossible to attribute to the old or the new record.
2. Confirm the deletion path still exists. Buttons move, API parameters get renamed, settings get relocated. Exercise the delete control before you rely on it, and note what you used.
3. Keep the record safe to find. Never plant real client data. This is the same discipline the wider vendor-diligence literature insists on: field-level scoping of what data a vendor touches, including prompts, outputs, metadata and logs, per Censinet. A canary that contains real personal data is a liability, not a test.

If you generate synthetic test data programmatically rather than by hand, keep the same discipline. The discussion of synthetic data in Lancet Digital Health is a useful caution: synthetic data is not automatically privacy-safe, and the assumptions baked into a generator travel with the output.

Run the new canary the same day you learn of the change, with a record created under the new terms. Do not wait for the old record's check date to arrive; by then you have already spent weeks measuring the wrong policy.

Then compare along four axes, not one:

The comparison rule is simple: a result is only a finding if the expectation, the surface and the method are all comparable. Anything else is a change in the test, and you label it as such.

The canary testing guidance from GrowthBook applies here even though it is about software releases: a canary asks whether a candidate violated explicit health criteria, and the comparison against a stable baseline is what gives the answer meaning. If your baseline moves because the criteria moved, you are no longer testing the same thing. The same principle runs through the general canary-deployment literature, which stresses that safety comes from containment and a rehearsed reversal rather than from hoping a small sample is harmless (Wingify). Your containment here is the record; your reversal is the ability to rerun cleanly.

When the check date arrives, search every surface you can reach. The canary guide lists them: in-product search and history, a full account export, memory and personalization prompts in a fresh session, API list endpoints for files, threads and vector stores, your own logs, and connected apps. A support request quoting the UUID and asking in writing whether any record containing it exists is worth more than it looks, because a written answer is evidence you can file.

Interpretation, with a decision rule: a hit inside the documented window is expected behavior, so close it. A hit after the window is a finding, so escalate in writing with the record. A clean result proves less than a hit, because it only tells you that the surfaces you can reach are clean.

The scheduling rule holds: set the interval to no more than half the new retention period. With a 90 day window, that means a canary every 45 days or sooner, so a change in terms is caught before two full windows pass without a test. With a 30 day window, run at least monthly.

That rule works because policy changes tend to arrive between your check dates. If your only canary was planted six months ago and the terms changed twice since, you have no comparable data for either change.

Add a second, cheap layer: a terms watch. Assign one owner to review the policy and DPA pages on a fixed cadence and on any notification, and log the review even when nothing changed. The vendor diligence literature is consistent that this is a monitoring job, not a one-time questionnaire: Captain Compliance frames AI vendor diligence as something that must connect to ongoing vendor management and monitoring rather than sitting buried in procurement, and the Initialize AI evaluation checklist names "no owner monitors the vendor after purchase" as a distinct failure mode.

The Optro questionnaire guidance makes the same point from the other direction: ask for retention and deletion policies in writing so they align with your own governance, which gives you the clause you later test. And the CTO Input due diligence checklist supplies the reusable rule for leaders: if it is not in the contract, it does not exist. A canary is how you find out whether what is in the contract is happening.

Two supporting practices make the schedule survivable:

• Keep the contract aligned with the test. If the DPA states a retention period, your canary should assert that period, and a written retention schedule with deletion dates for prompts, logs, backups and third-party copies is the baseline to test against, per Censinet. Public DPAs show how much variance exists in how those commitments get expressed: compare the OpenAI DPA, the Google Cloud DPA, the Datadog DPA and the Day AI DPA. Your canary tests the specific clause you signed, not the genre.
• Set a breach-notice tripwire in the same calendar. GDPR Article 33 requires controllers to notify the supervisory authority without undue delay and where feasible within 72 hours of becoming aware of a breach, and processors to notify controllers without undue delay (GDPR Art. 33). A breach notice is itself a trigger to rerun the canary, because data you believed deleted may now be in scope.

If you are instrumenting canaries across several providers, or across many models behind one interface, much of the effort is plumbing: identical plant, delete and search steps repeated for every surface. Consolidating model access behind a single OpenAI-compatible endpoint with prepaid US dollar pricing, such as Quanta Inference, reduces the number of credentials and surfaces you have to instrument. The test logic does not change; only the count of places it must run shrinks.

Six failure modes account for most bad canary conclusions. Check your rerun against all six.

The deeper insight is that a retention canary is a comparison instrument, not a compliance checkbox. Its value comes from holding the expectation fixed while the world moves. When the provider moves the expectation, the instrument has to be recalibrated before it can measure anything, and the recalibration itself is the evidence you will want the next time someone asks whether vendor terms actually mean what they say.

If your team would rather not own the calendar, the record format and the surface inventory, that is a good use of a free 30-minute discovery call: bring your current DPA clauses and the last canary result, and leave with a rerun plan and a schedule sized to your actual retention windows.