Gateway, Sidecar or In-Process Routing for Model Calls

Latency is the criterion teams argue about most, and it is the one where the three options differ least in steady state and most in the tail.

A sidecar adds a local network hop on the same host or pod. In-process routing adds nothing measurable beyond the CPU cost of evaluating the rule. A centralized gateway adds a network hop plus the gateway's own processing, which for a mature Go-based AI gateway is documented as 11 microseconds of overhead per request at 5,000 requests per second (Maxim AI). Against a model call that routinely takes hundreds of milliseconds, that overhead is noise.

The real latency risk in any enforcement point is cold start and scale-from-zero. Cold starts are the first-invocation latency a function pays after deployment, after a scale-from-zero event, or after the platform reallocates resources, and the duration depends heavily on function size, dependency count and runtime (PayPro Global). For a minimal hosting platform folded over an edge runtime, one vendor reported a median cold latency of 400ms and a median hot latency of 125ms, with regional variation of roughly plus or minus 100ms (Supabase discussion).

The decision rule: if your model call budget is in the hundreds of milliseconds, all three are viable and you should choose on ownership and blast radius. If your budget is tight enough that a cold start of several hundred milliseconds would break an interaction, in-process wins because it shares the application's lifecycle.

Region choice also interacts with latency in ways that surprise teams. Edge functions run closest to the user by default, but when a function makes intensive database calls, running it in the same region as the database usually performs better, and a pinned region is not automatically failed over during an outage (Supabase docs). A gateway that pins requests to a region inherits that same constraint.

Blast radius has two dimensions, and reviewers routinely conflate them. The first is failure: what happens to traffic if the enforcement component goes down. The second is compromise: what an attacker can do if that component is taken over (Trussed).

A gateway concentrates blast radius. It delivers the strongest fleet-wide consistency and a single place to change policy, and for exactly that reason its failure or compromise affects every calling service at once (Trussed). A careful gateway design mitigates this. Kong's NVIDIA Switchyard policy, for example, fails open by default: if the decision service is slow, down or returns something unusable, traffic is still served by a configured default target, and the policy can run in an observe-only mode where decisions are logged but never applied (Kong docs). That observe-then-enforce pattern is the single most valuable thing you can copy here.

A sidecar localizes failure to one workload, but only if it is deployed consistently. Coverage gaps are the characteristic sidecar risk: a pod without the proxy silently bypasses every policy (Trussed). The same applies to an embedded SDK, where enforcement depends on every service importing the SDK and staying on a current version.

In-process routing puts the failure domain exactly where the application's is. When the process is unhealthy, its policy checks are too, which is predictable but unbounded if one rule has a bug, because the bug ships with the service.

For residency specifically, none of these patterns lets you ignore the provider side. Residency means data, including in-transit copies, backups, metadata and processing intermediaries, stays within a jurisdiction for the whole lifecycle, and a provider that stores data in a US region but routes telemetry elsewhere may not satisfy a strict requirement (OneSource Cloud). IBM frames the same point as a sovereignty question: where data resides determines whose law governs it (IBM).

Vercel's AI Gateway documents the mechanics honestly. Setting inferenceRegion pins where the provider runs inference and stores data, and if the gateway cannot honor the chosen region, the request fails rather than silently running elsewhere. But it does not pin where your request terminates before forwarding to the provider (Vercel docs). That distinction, between pinning inference and pinning the gateway's own processing, is the one most compliance reviews miss.

// A residency pin that fails closed rather than silently rerouting.
const response = await gateway.chat({
  model: "provider/model-id",
  messages: [{ role: "user", content: prompt }],
  providerOptions: { gateway: { inferenceRegion: { scope: "zone", geoRegion: "eu" } } },
});
// If the EU zone cannot serve the request, this throws. Catch it and
// surface a policy error, never a retry into a different geography.

This criterion decides the question faster than any latency benchmark, because the answer is often "no."

An API gateway is language-agnostic by construction: services speak HTTP, and the gateway never inspects their runtime. That is why a gateway can enforce one allow-list across a Python service, a TypeScript agent and a Java backend without anyone agreeing on a library.

A sidecar is also language-agnostic, since the proxy intercepts traffic below the application, and mesh implementations typically handle HTTP and gRPC with some TCP support (Solo.io). The cost is operational: you now run a proxy per pod and need a control plane or an equivalent rollout mechanism to keep versions aligned.

In-process routing is where language support bites. You need an SDK for every runtime in your fleet, and you inherit its update cadence. If your fleet includes a long tail of languages, embedded enforcement will always have holes unless you mandate a supported set. Cloudflare Workers and Deno Deploy both run JavaScript, TypeScript or WebAssembly, and edge runtimes specialize in these languages rather than covering arbitrary runtimes (Stack Overflow, Vercel). The same narrowing applies to in-process policy: it works beautifully inside a homogeneous stack and awkwardly outside one.

A practical hybrid, and the one most teams land on within a quarter: enforce residency and cost caps at the gateway, where coverage is guaranteed, and let in-process code choose the model within the allowed set for latency-sensitive paths. That keeps the unbounded decisions bounded.

Ownership is the criterion that predicts success better than any technical measure, because policy that nobody owns drifts.

A gateway makes policy a platform-team artifact. The platform team holds provider credentials, routing rules, cost budgets and regional pins, and service teams call one endpoint without hard-coding a provider (Maxim AI). The gateway is also the natural place to hold the API keys, which means individual developers never handle production credentials for model providers. When a regulator asks who can send data to which country, the answer is one repository, one review process and one audit stream.

A sidecar splits ownership between platform and service teams. The platform team ships the proxy and the control plane, the service team declares its own rules, and rollout is independent per service (Trussed). This is attractive when a regulated business unit needs residency rules differing from the rest of the company, and painful when nobody can say which sidecar version is running in production.

In-process routing gives the policy to the application developer, in their repository, on their release cadence. That is the fastest loop for iterating on routing logic, content-based model selection or fallback behavior. It is also the model in which a developer can quietly route EU customer prompts to a US endpoint and no platform control notices. For cost caps, in-process is genuinely appealing: a content-based router choosing a cheap model for easy requests and a frontier model for hard ones can reduce cost substantially (Maxim AI), and that logic evolves fastest next to the prompt it operates on.

Use this tie-break when your team is divided: if a regulator, auditor or CFO would need to ask more than one team to get a complete answer, your enforcement point is wrong for your governance model.

Work through these in order and stop at the first "yes."

1. Does any single rule, residency, a cost ceiling or an allow-list, apply to more than three services? If so, an API gateway is the only option that guarantees coverage without depending on every team's build pipeline.
2. Do you need to prove the rule is enforced, with an audit stream a reviewer can follow? Centralized enforcement produces one audit stream naturally, while distributed patterns require aggregating logs from every instance (Trussed). Choose the gateway.
3. Does a regulated business unit need different residency rules from the rest of the company while sharing infrastructure? A sidecar gives per-service policy with a local hop and independent rollout.
4. Is your latency budget so tight that a cold start of several hundred milliseconds would fail the interaction? Move enforcement in-process, and keep the gateway as the outer bound for residency and cost.
5. Is your stack homogeneous in one runtime, and the policy simple (a model allow-list and a per-request token cap)? In-process is the cheapest to operate and the fastest to iterate.
6. If none of these is decisive, start with the gateway in observe-only mode, mirroring the pattern Kong documents where decisions are logged but not applied (Kong docs). After two weeks you will know from the logs whether in-process routing would have changed any outcome.

Two implementation notes that save pain. First, verify the region actually used rather than assuming it: Supabase exposes an x-sb-edge-region response header and logs the region, which turns a policy claim into a measurement (Supabase docs). Second, test the failure path deliberately. If your gateway fails closed on a region it cannot honor, your on-call runbook needs to say what the caller sees, and the observable headers or logs need to distinguish enforced, observed and fallback outcomes (Kong docs).

Model routing decisions also belong in one place when you run many providers. In a survey of enterprise technology leaders, 37 percent of respondents reported using five or more models, up from 29 percent a year earlier, so each additional provider multiplies the routing questions someone has to answer consistently (Maxim AI). That arithmetic is the strongest argument for the gateway when your model count climbs.

If you want a single endpoint across text, image, video, audio and embedding models while your gateway holds the policy, Quanta Inference exposes every model kind through one OpenAI-compatible API key, prepaid and priced in US dollars, at https://quantaintelligence.ai/inference. Keep the gateway as the place residency and cost caps are enforced, and let it forward to one upstream.