A Residency Rule Inside a Model Gateway: Schema and Cache

Define one row per permitted route. A route is the tuple of tenant class, purpose, model and region that the gateway may send traffic to. Anything not in the table is denied.

Two fields carry more weight than they look. inference_region maps to a real gateway parameter. Vercel exposes providerOptions.gateway.inferenceRegion with zone values us, eu, au and jp, and zone requires a geoRegion or the request fails Vercel. Note what Vercel also says: the setting pins where the provider runs inference and stores data, but not where the request terminates before it is forwarded, and region-pinned gateway hosts that terminate TLS in a single region were, at the time of writing, listed as coming Vercel. If your tier is end-to-end, that distinction is the difference between passing and failing an audit, so record it in the row.

retention_days is the field teams forget. An LLM request path duplicates data at every hop: prompt logs, traces, eval datasets, vector stores, and vendor-side retention you did not model Kunal Ganglani. A cached response is one of those copies. If the route's retention is zero days, the cache entry's lifetime is zero, which means the route does not cache at all. The table decides that, not a constant in the cache client.

How the table behaves at runtime:

1. The gateway authenticates the request and derives tenant_class and purpose from the API key and the request metadata, never from a client-supplied header.
2. It looks up the matching row for policy_version. No row means deny, with a distinct error code so callers can tell policy denial from provider failure.
3. It passes inference_region to the provider call and records the vendor and tier in the trace.
4. It passes retention_days and policy_version to the cache layer.
5. It writes one audit record per call: tenant class, purpose, vendor, region, policy version, cache outcome.

Amazon Bedrock illustrates why step 3 needs care. Bedrock's cross-region inference offers two profile types: geographic profiles that keep routing within a defined geography (US, EU, APAC, Australia, Japan), and global profiles that route to supported commercial regions worldwide for throughput and carry a lower per-token rate hidekazu-konishi.com. The same mechanism serves both throughput and residency, and choosing the wrong profile type silently changes where data goes. In a gateway, that choice belongs in the vendor and inference_region columns, not in application code.

The decision rule

Write it once and apply it everywhere: a request is served only by a row that matches its tenant class, purpose and policy version, and the row's region is passed to the provider as a hard pin, not a preference. If no row matches, the request fails. If the provider cannot honor the pin, the request fails. Fallbacks are allowed only to another row with the same tenant class, purpose and region, and a different vendor. A fallback that crosses a region boundary is a policy violation with a retry loop attached.

Caching LLM responses is a real cost lever, and it comes in layers. Exact-match response caching hashes the whole request; semantic caching embeds the request and serves a near-identical prior answer when similarity clears a threshold TrueFoundry. Both are useful. Both are dangerous in a residency-controlled gateway, because the cache is a data store and it inherits the residency and retention rules of whatever produced the entry.

The leak shapes are documented. A semantic cache can serve one user's response to another because two embeddings landed within a small cosine distance, and the cache is working exactly as designed when it does so Tianpan. The listed failure modes include authorization scope collapse, where a personalized answer like "show me my open tickets" is cached on the query's embedding alone, and entitlement bypass, where a free-tier user receives a paid-tier answer because access tier was not in the key Tianpan. Residency is the same class of context: the embedding of a prompt does not know which region is allowed to answer it.

So the key must carry that context explicitly. Build it from five segments:

import hashlib, json

def cache_key(tenant_id: str, purpose: str, prompt: str,
              policy_version: str, model: str) -> str:
    prompt_hash = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
    material = json.dumps({
        "tenant": tenant_id,          # never a tenant class; the actual tenant
        "purpose": purpose,
        "prompt": prompt_hash,
        "policy": policy_version,     # from the table, read per request
        "model": model,
    }, sort_keys=True)
    return "llm:" + hashlib.sha256(material.encode("utf-8")).hexdigest()

Three properties fall out of this key, and each one is a bug you avoid:

• Tenant isolation is structural. Two tenants asking the identical question produce different keys. No similarity threshold can collapse them.
• A policy change invalidates everything. Bump policy_version and every prior entry becomes unreachable. Nothing has to be purged on a schedule for correctness; old entries age out and are deleted for storage hygiene.
• The model is part of identity. Swapping the model behind a route changes the answer, so it must change the key.

The policy version is the segment that makes residency testable. Without it, consider a concrete case: a route for eu-health support replies was pinned to an EU region and cached under policy_version res-3. Someone edits the table so the route's vendor changes to one whose EU endpoint stores data at rest outside the EU. The route is now non-compliant, but every cache hit continues to serve an answer generated under the compliant configuration, with no indication that the serving path changed. With the version in the key, the first request after the table changes misses, goes to the new vendor, and the audit trail shows the transition at the exact request where it happened.

For semantic caching, the same rule applies one level deeper: namespace the vector index by tenant and policy version, and refuse to compare across namespaces Tianpan. The similarity threshold then governs recall within a single tenant's own history, which is a quality question, not a residency question. The conservative approach recommended for production is to keep semantic caching for public or low-sensitivity routes and exact-match caching everywhere else TrueFoundry.

Retention is a cache setting, not a storage setting

An entry's time to live should be the minimum of the route's retention_days and whatever quality argument you have for caching at all. If retention_days is zero, the write is skipped. This keeps the cache inside the policy rather than beside it. It also matters for the broader data path: if you log full prompts in production, you have built a shadow database, and the same is true of a response cache that no one classified Kunal Ganglani.

A support assistant serves two tenant classes: eu-health, with a processing-residency requirement, and us-generic, with storage residency and a contractual 30-day log retention. The gateway receives 1,000,000 requests a month, split 300,000 for eu-health and 700,000 for us-generic. Exact-match caching hits on 40% of requests in both classes.

Under policy_version res-3:

• eu-health sends 300,000 requests. The cache serves 120,000 and 180,000 reach the EU-pinned provider. Cache entries carry retention_days of 7, the shortest value any reviewer would sign off, so entries expire in a week regardless of hit frequency.
• us-generic sends 700,000 requests. 280,000 are cache hits, 420,000 reach the provider. Entries live 30 days to match the log retention.

Now the table is edited: the eu-health support route moves to a different vendor, and policy_version becomes res-4. Two things happen without anyone touching the cache service. First, all 120,000 former hits for eu-health become misses until new entries are written under res-4, so roughly one month's worth of requests pay full price once while the cache refills. Second, the audit log shows every one of those misses at the request where the vendor changed, which is exactly the evidence a reviewer needs. Had the key omitted the version, all 120,000 hits would have been served from entries generated by the old vendor, and the switch would have been invisible in the data plane.

The cost of getting this wrong in the other direction is worse. Suppose the same edit accidentally changed eu-health from a processing-residency pin to a global routing default. With a versioned key, the next miss goes out under res-4 and the gateway can be made to reject it, because a processing tier row with a global region is an invalid row. The validator refuses to publish res-4. Without versioning, you would need a cache flush coordinated with a routing change, and any gap between them is a residency violation that no dashboard reports, because a cache hit looks like a healthy 200.

A replay suite that runs against a hard-coded list of providers tests a system that does not exist. The fix is unglamorous: the replay harness imports the same policy table module and the same validator the gateway uses, and every recorded request is resolved through it before the model is called.

What the replay gate should assert, per recorded case:

1. Resolution. The case's tenant class and purpose resolve to exactly one row at the candidate policy_version. A case that resolves to zero rows fails, and a case with an invalid tier and region combination fails.
2. Region pin. The resolved row's inference_region is what the harness passes to the provider, and the provider's response metadata confirms it. The gateway behavior to imitate is Vercel's: an unhonorable pin fails the request rather than being retried elsewhere Vercel.
3. Cache key identity. The harness computes the key the gateway would compute and asserts it differs when tenant, model or policy_version changes. This is a unit test with no model call in it, which makes it cheap enough to run on every commit.
4. Fallback boundary. If the route has fallbacks, assert that every fallback resolves to the same tenant class, purpose and region. A fallback that crosses regions fails the build.

The comparison that makes the case for this gate is a table of two review styles.

The last row is the quiet payoff. Because every call writes its policy_version, vendor, region and cache outcome, you can answer "which routes served eu-health traffic in the period after the table changed" by querying logs rather than by interviewing engineers. Gateway tracing already gives you the raw material: request logs, spans, and per-call attributes exist to explain what the gateway sent to the provider and why a call was slow or wrong TrueFoundry. Adding policy version to that record is a field, not a project.

Two operational habits keep the gate honest. First, the policy table should be reviewed like code, with the validator running in CI, because the validator is the thing that turns a residency promise into a build failure. Second, the replay corpus should include the cases that used to be edge cases: a request from each tenant class, at least one case per tier, and one case per route whose vendor has changed at least once. When the replay suite runs through the table, the table stops being documentation and starts being a test fixture.

If you are wiring this up across several providers, the routing surface matters less than the policy surface. Residency rules that live in the gateway are defined once for all traffic, while rules that live in application code get duplicated in every service that calls a model Maxim AI. The table, the validator, the versioned cache key and the replay gate are portable across gateways. Models and endpoints are not. If your stack already spans several model families and you would rather not run one gateway per provider, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API, and you can keep the policy table and cache key design in front of it. For teams that want the routing and replay work built with them rather than bolted on afterward, Quanta pairs you with a specialist who builds the app and the surrounding policy.