A Golden-Set Check That Blocks a Gateway Rule Change

Do not pull a uniform random sample. Uniform sampling over-represents your top three intents and under-represents the long tail that breaks first. Stratify on the signals the rule can act on: route or endpoint, served model, tenant or plan tier, region, request type, and prompt length bucket. Then sample per stratum.

The floor of 30 per stratum is a heuristic, not a law. At 30 items, one flipped result moves the pass rate by about 3.3 percentage points. That is too coarse to resolve a 1 point threshold, so apply the 1 point rule to whole suite metrics, where 500 or more items gives you roughly 0.2 points of resolution per item, and use per stratum counts as a smoke check for collapses.

Size the suite against the smallest regression you care about. To see a 1 percentage point absolute drop with any confidence, you want on the order of 500 to 1,000 replayed items, and the right number depends on your baseline rate and tolerated noise. Measure it instead of guessing: run the current config against the suite twice with no change and look at how much the pass rate moves. That spread is your noise floor, and the block threshold must sit above it.

Do not forget the error strata. A routing policy that fails open is easy to ship and hard to notice, and Kong's NVIDIA Switchyard policy documents exactly that pattern: it falls back to a configured default_target on an unknown target, drift, timeout or error, and logs a single JSON line with the reason (Kong docs). If your suite contains no items that trigger fallbacks, you have tested the happy path and nothing else.

Every item needs four things: the request exactly as the gateway sees it, the headers that influence routing, the expected output or assertion that decides pass, and provenance (when captured, from which stratum).

Capture at the gateway rather than in the application. Most gateways emit OpenTelemetry generative AI semantic conventions, so you get token usage and model identity without a custom exporter. Agentgateway exposes a token usage histogram labeled with gen_ai_token_type, gen_ai_system and gen_ai_request_model, and normalizes input counts so a field means the same thing across providers: Anthropic and Bedrock exclude cached tokens from reported input, while OpenAI, Azure and Gemini include them (agentgateway observability). Normalization matters for replay, because raw counts are not comparable across a routing change.

Redaction happens before storage, not at replay time. A replay dataset is exactly the "temporary" prompt log that becomes the most sensitive system of record you run. The LLM data leakage playbook describes the lifecycle: replay sets start as a sample, become useful, become relied on, become permanent, and end up as a customer text archive with no deletion story (Kunal Ganglani). Two controls do most of the work:

• Use a maintained anonymizer rather than hand-written regexes. LangSmith's create_secret_anonymizer redacts recognized credential formats, including provider key prefixes, cloud access key IDs, GitLab and Slack tokens, PEM private keys, authorization headers and credentials embedded in URLs, replacing each match with a placeholder before upload (LangChain docs). Read the stated limits: it covers inputs, outputs, errors and extra metadata, while run names, tags and attachments pass through unmodified.
• Replace personal data with stable tokens rather than deleting it. Swapping an email for [[EMAIL_1]] consistently keeps the item realistic and keeps the baseline and candidate comparison fair.

Keep the routing headers. If a rule branches on a tenant header or an API version header, a suite that strips headers replays a different request than production sent, and your gate measures the wrong thing.

Each item needs a deterministic verdict before you replay anything. Rank graders by reliability:

1. Exact or schema match for structured output: required keys present, types correct, enums valid. This is the strongest signal and should cover most extraction and classification traffic.
2. Semantic match for free text: an embedding similarity threshold, or a judged rubric with a pinned judge model and pinned prompt.
3. Human labels as ground truth on a small subset, used to validate the judge.

If you use a judge model, pin it and pin its prompt in the repository, then run it through the same path for baseline and candidate. A judge that silently upgrades destroys the gate, because you can no longer tell whether a drop came from the gateway change or from the grader.

Store per item grader output, not just aggregates. When the gate blocks, you need to open the failing items and see why.

Replay sends each recorded request through the gateway with the candidate config applied, captures the response, and grades it. Three rules make the comparison valid.

Run baseline and candidate back to back on the same day against the same suite version. Provider load and model side variance move results, and a baseline from last week is a different measurement.

Keep harness concurrency and retry behavior identical across both runs. If the candidate config changes retry counts or timeouts, that is part of what you are testing, but the harness must not differ between runs.

Replay in staging with production-like credentials and quotas, and include items that exercise the failure paths. Residency pins matter here too: Supabase returns an x-sb-edge-region header so you can verify where a function actually executed, and its docs warn that when you pin a region explicitly, requests are not automatically re-routed during an outage (Supabase regional invocations). That warning belongs in your suite as an assertion, not in a wiki page.

A minimal harness shape:

baseline_cfg = load_config("gateway/current.yaml")
candidate_cfg = load_config("gateway/candidate.yaml")
suite = load_suite("evals/golden_set_v7.jsonl")

def run(config, suite):
    results = []
    for item in suite:
        resp = replay(item.request, gateway_config=config)
        results.append({
            "id": item.id,
            "stratum": item.stratum,
            "pass": item.grader(resp),
            "latency_ms": resp.elapsed_ms,
            "served_model": resp.headers.get("x-served-model"),
            "region": resp.headers.get("x-region"),
            "fallback": resp.headers.get("x-routing-outcome"),
        })
    return summarize(results)

base = run(baseline_cfg, suite)
cand = run(candidate_cfg, suite)
assert_gate(base, cand)

Capture the served model, region and routing outcome from response headers on every item. Kong's policy returns headers naming the target that served the request and whether the outcome was enforced, observed or fallback, which turns "quality dropped" into "quality dropped because 12 percent of traffic moved to the fallback target."

The gate is deliberately blunt. For every quality metric (overall pass rate plus pass rate per critical stratum), compare candidate against baseline and block on three conditions.

• Any quality metric drops by more than 1.0 percentage point absolute versus baseline.
• Suite p95 latency rises more than 15 percent versus baseline, measured on the same suite and the same run window.
• Any residency assertion fails, such as a request pinned to a region being served elsewhere.

Here is why 1 point is the right shape of threshold. A baseline of 94.0 percent on 1,000 items and a candidate at 92.7 percent is a 1.3 point drop, about 13 items flipped, and the gate blocks. A candidate at 93.5 percent is a 0.5 point drop on 5 items, inside the noise band you measured, and it passes. The threshold turns an argument into a count.

The 15 percent latency rule works the same way. A baseline p95 of 900 milliseconds allows up to 1,035 milliseconds before blocking. Compare that with the placement effects people actually report: Supabase's medians varied by roughly plus or minus 100 milliseconds across regions, and its docs note that a function doing multiple database round trips is often faster placed near its data than near its user. A region change can plausibly cost more than 15 percent, which is why you measure instead of assuming.

Keep cost advisory at first. Many gateways compute realized cost per request from a model cost catalog and expose it in logs and traces along with token counts, which makes the check cheap to add. But cost and latency trade against quality in ways the gate should surface rather than silently forbid. Leave it as a pull request comment until the team has run it for a cycle.

The CI job keeps three versioned artifacts: the suite, the baseline results and the candidate results. A red gate should print per item diffs and the routing distribution across targets, so the author can see which rule did the damage.

A team proposes sending short prompts to a cheaper model and keeping long prompts where they are. They sample 800 production calls across five intents and pin 20 items per region across two regions.

Baseline on the current config: 752 of 800 pass (94.0 percent), p95 880 milliseconds. Candidate: 741 of 800 pass (92.6 percent), p95 812 milliseconds. Latency improves, so no block there. Quality falls 1.4 points, which blocks.

The per stratum breakdown decides the outcome. The 1.4 point aggregate hides a 9 point drop in the "billing dispute" stratum, 35 items, offset by a 2 point gain on "order status." The rule classified some billing disputes as short prompts, sent them to the cheaper model, and the answers failed the rubric. The author narrows the rule to exclude that intent, re-runs the suite, and the next candidate scores 93.6 percent with 1 point of latency improvement. It merges.

That is the insight a first search result will not give you: a healthy global pass rate can hide a catastrophe inside one routed segment. Always evaluate per stratum, and treat any large per stratum drop as a block even when the aggregate looks flat.

A replay suite decays. Intents disappear, prompts change, and models behind the gateway move. Four habits keep it useful.

Recapture on a schedule with the same stratification rules and replace a fixed share of items each cycle. Re-measure the noise floor every cycle by running the current config twice with no change. Delete items whose expected output was wrong and log why, because silent edits to golden data are how a gate stops meaning anything. Review residency assertions whenever a provider or region is added, since a new route is the easiest place for a pin to go missing.

If the harness needs to replay text, image and embedding calls through a single OpenAI-compatible endpoint, Quanta Inference (https://quantaintelligence.ai/inference) can sit behind it, with prepaid billing in US dollars.