How to Test a Model Provider's Data Retention Through Its API

A canary is a digital tripwire that has no legitimate function: if it shows up, the fact of its appearance is the evidence the Grafana Labs write-up and the Acalvio glossary. For a retention test the canary has to satisfy four rules.

A worked example. Generate canary-7f3a9c2e41b84d0ea6f5c1d2b9e0477a and tag it with your test date in a separate variable, not inside the token itself, so the token stays stable if you re-run the test. Record the token in your test log before sending anything. If your test later surfaces the token anywhere, you can identify exactly which request created it, because no other request in the organization will ever contain that string.

One rule worth stating explicitly: do not use a canary you cannot afford to see again. If a provider retains content for a year under a legal-hold carve-out, your canary will sit there for a year. Synthetic strings are fine for that; real data is not. This is the same principle as the pii handling advice in the improved-API discussion, which recommends scrubbing before sending so sensitive data never leaves your server.

The mechanics are ordinary API work, which is the point: the test should be boring and repeatable. Send each canary in a request, capture the full response (status, headers, response id, usage fields), and store both in a dated folder.

import json, time, uuid, hashlib, pathlib

def new_canary(account: str) -> str:
    raw = f"{account}:{time.time()}:{uuid.uuid4().hex}"
    return "canary-" + hashlib.sha256(raw.encode()).hexdigest()[:32]

def send_probe(client, canary: str, placement: str, model: str):
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": f"Repeat this token back exactly: {canary}"}],
        "metadata": {"placement": placement, "test": canary},
    }
    started = time.time()
    resp = client.chat.completions.create(**payload)
    record = {
        "canary": canary,
        "placement": placement,
        "request_id": getattr(resp, "id", None),
        "sent_at": started,
        "response_text": resp.choices[0].message.content,
    }
    pathlib.Path("probes").mkdir(exist_ok=True)
    pathlib.Path(f"probes/{placement}-{canary}.json").write_text(json.dumps(record, default=str, indent=2))
    return record

Three placements are worth testing separately: canary inside a normal user prompt, canary inside a system or developer message, and canary inside an uploaded file if the provider offers file or document input. Keep each in its own probe so a hit tells you which surface retained it. Also send one benign prompt with no canary, as a control, so that if a search later returns a fuzzy match you can tell whether it is your canary or ordinary text.

Capture the response id. Many providers surface request ids in their dashboards and support tooling, and that id is the bridge between your test log and whatever the provider can look up later. If you are testing a wrapper or proxy in front of the model, run the probe against the wrapper and against the direct provider endpoint, because the two hops may have different retention behavior, as the multi-hop discussion notes the dev.to analysis.

A word on what not to test: do not send canaries designed to trip content classifiers unless that is a deliberate part of your plan. Flagged content is exactly the category providers carve out for extended review, so a canary that trips a filter tests the exception, not the rule. If you do want to test it, make it a separate, clearly labeled probe whose result you interpret against the safety-retention clause rather than the default deletion window.

The hardest part of this test is not sending the canary. It is deciding, in advance, which surfaces you are checking and which count as a hit. Without that decision, a search after the fact becomes an argument about what you found.

Make a list before the first probe:

Two of these need commentary. First, your own logs are a surface you control, and a hit there is a finding about your code, not the provider. Teams discover they log full request bodies at info level and keep them for weeks longer than they imagined, which is a real risk and the kind of thing a test should catch the CTO Input checklist and the Initialize AI vendor checklist both treat logging and export evidence as part of vendor review, and the same discipline applies internally.

Second, the provider's usage dashboard and audit logs are often the only observable surfaces for a customer. What matters is that you check where you can, name the surfaces you could not check, and say so in the result. A test result that says "canary absent from provider dashboard and audit export at day 31, not observable in support tooling" is honest and useful. A result that says "provider deletes data" is not, because it was never something the test could show.

Before you begin, write down the number. If the DPA says 30 days, record the exact retention quote and the section it came from. Open retention statements are common: a provider may say it retains data for abuse monitoring without putting an explicit outer bound in the customer-facing terms. Write that down too.

The test has four timestamps: send time, response time, promised deletion time (send time plus the stated retention period), and recheck time. The recheck should happen after the promised deletion, with a small buffer, because backup copies and log pipelines often lag.

A practical schedule for a 30-day promise:

T+0    Send all probes. Save responses, request ids, and the quoted retention clause.
T+7    Check dashboards and any available exports. Expect a hit; this validates the search method works.
T+31   Recheck every observable surface. Canary should be gone.
T+38   Recheck again to catch backup and pipeline lag.
T+61   Optional final pass if the provider has a longer secondary retention for safety or backups.

The T+7 check is not optional and is the most common mistake in this kind of test. If your search cannot find the canary while it is definitely still there, then a miss at T+31 proves nothing, because your search never worked. Validate the search first, on a surface where you know the token exists, then trust the later misses. This is the same validation logic as any canary-token deployment: the alert has to be known to fire before a quiet period can be read as safety the Acalvio glossary.

At each checkpoint, record the exact query, the surface, the date and time, and the result. If you found the canary, capture the row and the context around it. If you did not, capture the query text and a screenshot or export showing zero results. The evidence value is in the negative result being demonstrably a real search, not an unremembered one.

For providers that quote a different window for different categories, schedule each category separately. If the DPA says 30 days for content and longer for abuse-monitoring records, two recheck dates and two classifications are needed. Handling each retention clause as its own test line prevents a single "passed" from covering a clause you never checked.

Sort every result into one of four outcomes.

A hit does not automatically mean the vendor lied. Safety and abuse-monitoring retention is a real and disclosed category, and a canary sitting in an abuse-monitoring log past the content window may be exactly what the DPA describes. The test's job is to tell you which category the hit falls into, using the canary placement and the timestamp. A canary that shows up in a dashboard you check daily is a different finding from one that appears in an export you ran at day 61.

The decision rule is what matters before signing: any unresolved Fail result is a reason to hold the signature or to renegotiate the retention terms, not a reason to look harder for an excuse. The compliance checklists converge on the same point. "Paper beats promises," as one healthcare-focused checklist puts it, with retention schedules and no-training terms written into the contract, plus breach deadlines in hours rather than "promptly" the Censinet checklist. The vendor evaluation checklists frame the same requirement as evidence: retention policy, DPA, and logging documentation, not assurances the Optro questionnaire guide and the Captain Compliance review.

If a hit is real and unclarified, the next step is a written question to the vendor, with the request id and the timestamp from your test log. Ask which retention category the record belongs to, what triggers deletion from that category, and whether the DPA's stated window applies to it. A provider that answers precisely and provides an export showing the deletion is giving you something you can put in a risk register. A provider that answers vaguely has told you something useful too.

For the broader question of what to do once you have the test results, a free discovery call is the natural next step if you are evaluating a provider for a regulated or contract-sensitive use case a consulting call. The method here is doable internally, but interpreting the results against GDPR Art. 33 obligations, which require breach notification to a supervisory authority within 72 hours of becoming aware the GDPR text, and against your own DPA terms is where a second set of eyes often pays for itself.

The output of the test is an evidence pack you can attach to a vendor file, reuse at the next renewal, and show to an auditor or customer. A workable structure:

canary-retention-test/
  README.md          # scope, provider, date range, method, who ran it
  clause.md          # verbatim retention clause and section number from the DPA or terms
  probes/            # one JSON per probe: canary, placement, request id, response
  checks/
    2026-03-02-t07.md   # query text, surface, result, screenshot path
    2026-03-27-t31.md
  result.md          # pass, open, explained, or fail, with the reasoning

The README should name the exact surfaces checked and the ones that were not observable, and the result should restate the decision rule. Two things make this pack hold up: the retention clause is quoted verbatim with its location, and the search that produced each negative result is reproducible from the saved query. Everything else is convenience.

The same pack is useful at renewal. Providers change terms, add subprocessors, and adjust retention settings. A one-off test tells you about the system on the day you ran it; a dated pack plus a re-run at the next contract anniversary turns a single observation into a monitoring practice. The vendor evaluation guidance is explicit that approval is not the end: terms, models, and data usage change, and approved vendors still need review the Initialize AI vendor checklist. A canary test on a schedule is one concrete, low-cost way to do that review with evidence instead of memory.

If your test runs across several provider APIs and you would rather not maintain a separate SDK per vendor, routing the probes through one OpenAI-compatible endpoint keeps the harness identical across providers so the only variable is the provider's behavior Quanta Inference. Send the same synthetic canary, capture the same response shape, and compare results side by side.