Routing Model Calls Across Regions Without Splitting Your Data

Start by deciding what a jurisdiction tag actually is. It is not the caller's IP country. IP geolocation is a routing hint, not an entitlement, and treating it as one means a VPN changes which law applies. The tag should come from something authoritative: the tenant record, the data classification of the payload, or a header set by an authenticated upstream service that has already resolved the tenant.

A workable header contract:

POST /v1/chat/completions
x-jurisdiction: eu
x-tenant-id: acme-de
x-request-id: 01J8Z3K9QW

Three rules make the tag trustworthy.

First, the tag is mandatory. A missing tag is a rejected request, not a default. Silent defaults are how residency violations happen, because the gateway guesses and the guess is invisible in the logs.

Second, the tag is derived server side where possible. If your control plane already knows that tenant acme-de is contracted under EU terms, derive eu from the tenant record and ignore any client-supplied value that disagrees. Reject the mismatch and log it as a violation attempt.

Third, the tag is coarse. You need a jurisdiction, not a country and not a data center. eu, us, au, jp style zones are enough for policy, and Vercel's gateway uses exactly this vocabulary of us, eu, au and jp zones in its inferenceRegion reference. Coarse tags survive regulation changes better than fine-grained ones, because a new member state usually falls inside an existing zone.

If you need the tag at the infrastructure edge rather than in application code, remember that edge runtimes are deliberately constrained. The guide to edge runtimes from ResumeLens notes that edge functions run in V8 isolates, cannot read the filesystem, cannot spawn processes, and time out after a few seconds. A tag lookup at the edge should therefore be a cache read or a header inspection, not a cold database call. If you need to look up a tenant's jurisdiction from durable storage, do it in a regional service, not in a globally distributed isolate.

The policy table maps each jurisdiction to the endpoints that may serve it. It is the artifact auditors will ask to see, so treat it like a schema with review requirements, not like a config file that lives in a repository branch.

Two design decisions matter more than the rest.

First, fallbacks must stay inside the jurisdiction. It is tempting to write a fallback chain that ends with "any working endpoint," but that last hop is exactly where the compliance failure lives. Resolve fallbacks within the set of endpoints whose jurisdiction matches the tag and whose endpoint_class matches the requested capability. If none is healthy, fail the request. Vercel's gateway takes the same position: per its regional inference documentation, if it cannot honor the chosen region, the request fails instead of silently running somewhere else. That is the correct failure mode, and you should copy it.

Second, an explicitly pinned region does not fail over on its own. Supabase's regional invocation docs warn that when you specify a region with the x-region header, requests are not automatically re-routed to another region during outages, and that you should consider temporarily changing the region yourself. That is a feature for residency work: an outage should page a human, not quietly move regulated data to another continent.

A minimal table as code, suitable for review in a pull request:

POLICY = {
    "eu": {
        "support-chat": [
            {"endpoint_id": "eu-central-inference-a", "region": "eu-central-1", "class": "frontier"},
            {"endpoint_id": "eu-west-inference-b",   "region": "eu-west-1",   "class": "frontier"},
        ],
        "ticket-summary": [
            {"endpoint_id": "eu-central-small-a", "region": "eu-central-1", "class": "small"},
        ],
    },
    "us": {
        "support-chat": [
            {"endpoint_id": "us-east-inference-a", "region": "us-east-1", "class": "frontier"},
        ],
    },
}

The table is small on purpose. Jurisdictions and workloads multiply, classes do not. Keeping the vocabulary tight makes the table legible to a reviewer who is not an engineer, which is who usually has to approve it.

The routing step is short. Read the tag, look up the workload, filter to allowed endpoints, pick one using a deterministic rule, forward the call.

def route(jurisdiction: str, workload: str, policy: dict, health: dict):
    candidates = [
        ep for ep in policy.get(jurisdiction, {}).get(workload, [])
        if health.get(ep["endpoint_id"], {}).get("up", False)
    ]
    if not candidates:
        raise ResidencyError(
            f"no healthy endpoint for jurisdiction={jurisdiction} workload={workload}"
        )
    return candidates[0]

The deterministic rule matters for auditability. Pick the first healthy endpoint in policy order, not the lowest-latency one, unless you record the selection reason. A load balancer that spreads calls across several compliant endpoints is fine, but the audit record must say which one served each call, otherwise the log answers "which regions might have served this" rather than "which region served this."

Watch the layer where the region actually applies. Vercel's documentation is unusually candid here: setting inferenceRegion pins where the provider runs inference and stores data, but per the same page, it does not pin where the request reaches the gateway, which can terminate and be processed in any Vercel region before being forwarded. If your threat model covers the gateway's own processing, including prompt logging, you need region-pinned gateway hosts as well as region-pinned inference. Kong's AI routing provider policy documentation offers a useful contrast: the decision service can only select from targets you have already configured, so it cannot introduce a provider or URL you did not authorize.

There is a latency cost, and it is worth modeling before you promise anything. If a support chat in Frankfurt with three retrieval calls is served by an endpoint in us-east-1, the user pays for the round trips. One Supabase user reported their simple database-backed function going from about 250ms in us-west to roughly 1.1s in us-east and about 2.3s in se-asia, in a public discussion thread. Supabase's own maintainers later reported, in another thread, a median cold latency of 400ms and a median warm latency of 125ms for edge functions. Those numbers are for edge functions rather than model inference, but the shape transfers: crossing an ocean on every hop multiplies, and model calls add their own cold-start class of delay.

Cold starts deserve their own line in the capacity plan. The ACM systematic review of cold start latency in serverless computing catalogues caching and application-level optimizations as the main mitigation families, and a 2024 IEEE paper on dynamic pre-warm strategies reports reducing cold starts by up to 60 percent and improving average response times by 45 percent with prediction-based pre-warming. Residency narrows your warm pool to one jurisdiction, so pre-warming needs to happen per jurisdiction, not globally. A generic explainer on cold start latency puts Lambda cold starts between roughly half a second and a few seconds depending on function size and runtime, which is a useful order of magnitude when you are sizing a per-region warm pool.

The audit log is the deliverable that turns a routing trick into a control. Write one immutable record per attempt, including rejected attempts and failed calls, and include enough fields that a reviewer can reconstruct the decision without reading code.

Four practices keep the log defensible.

Write before you forward, or at least write a pending record. A crash mid-call should leave evidence that the call was attempted and to where.

Record the region from the response, not from the request. If the provider reports the region it used, in a response header or in body metadata, capture it. Supabase, for instance, exposes the execution region in its x-sb-edge-region response header and in function logs, per the regional invocation documentation. Kong's policy emits X-AI-Routing-Backend and X-AI-Routing-Outcome headers, documented in the same Kong page, which shows the pattern: the router tells you what it decided in the response.

Store the log in a region that satisfies the strictest jurisdiction it contains, or split logs per jurisdiction. A single global log that holds EU prompt metadata recreates the problem you just solved.

Never log the prompt body in the audit log. Log the shape of the call, not its contents, and keep content retention under the same jurisdiction policy as inference.

On the archive: keep audit records long enough to satisfy the strictest applicable requirement, and treat retention as a policy row you can point at. Vercel's comparison of zone-pinned versus global inference, covered in its regional inference page, is a good reminder that the global option exists and is cheaper; choosing it for a workload that does not need pinning is a legitimate, documented choice, as long as the choice is written down. For a broader view of how residency obligations accumulate across regimes, the U.S. data residency guide walks through HIPAA, state privacy laws and FedRAMP, and makes the point that residency covers backups, metadata and processing intermediaries, not just the primary store.

Take a support product with tenants in Germany, Australia and the United States. The table has three jurisdictions. A request arrives with x-jurisdiction: eu for support-chat.

The gateway derives eu from the tenant record, matches, and finds two frontier endpoints in eu-central-1 and eu-west-1. The first is unhealthy, so policy order selects the second, which answers in eu-west-1. The audit record names eu-west-1, latency 412ms, outcome served.

The next request carries x-jurisdiction: us for the same workload. It routes to us-east-1, is served, and is logged with us-east-1. The third carries no tag. It is rejected with a 400, logged with outcome rejected, and the on-call dashboard shows a spike in rejections. No data moved.

Now suppose the eu-west-1 endpoint also goes down. Policy order has no remaining healthy EU endpoint, so the request fails. That failure is the designed behavior, and it should page someone. The alternative, silently serving from us-east-1, would be invisible in every dashboard that tracks success rate. Supabase's note that pinned regions do not re-route automatically during outages describes exactly this tradeoff: availability versus boundary integrity, resolved in favor of the boundary.

If you would rather not build and operate this layer yourself, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the client contract stable while the routing policy underneath changes.

A residency gateway is a small system with a long tail of operational questions.

Policy changes. Every table edit ships with a policy_version and a reviewer. Ship the new version to shadow mode first: evaluate it against live traffic, log what it would have routed, and compare with what actually served the call. Kong's routing policy documentation describes starting in observe_only mode, where decisions are logged but never applied, as a first step. Copy that pattern even if you are not using Kong.

Region selection by the caller. Do not let clients choose a jurisdiction arbitrarily. Derive it, and compare. When the client value and the derived value disagree, reject and log. Over time the rejection log becomes a map of integrations that need fixing.

Same-region processing. Edge platforms that run code near the user are convenient, but as the Stack Overflow writeup on edge function infrastructure explains, the code executes wherever the network resolves the request, and that is the point of the design. If your gateway's own processing counts as processing under the regulation, run it in a pinned region and accept the extra hop. Imperva's explanation of origin and edge servers is a useful mental model: the edge is a cache and a filter, the origin is where the work you must govern happens.

Testing. Maintain a matrix test that, for every jurisdiction and workload, asserts that the set of reachable endpoints equals the allowed set. Run it in CI, and include a negative test for the missing-tag case, the mismatch case, and the all-endpoints-down case.

Review cadence. Once a quarter, diff the audit log's distinct serving_region values against the policy table's provider_region values. Any region in the log that is not in the table is either a bug or an undocumented exception, and both need an owner. A comparison like this catches the failure class the TrueFoundry gateway comparison warns about: gateways whose guarantees are enforced in some paths and not others.

Capacity per jurisdiction. Track warm capacity by jurisdiction. A workload with 95 percent of traffic in one zone still needs a warm pool in every zone where it can be served, and the pre-warm research cited above suggests prediction helps, but only if the prediction is per region.