Feature Flags for Model Swaps: Pinning and Canaries

A prompt and a model are a pair. A prompt tuned for one model's instruction-following style can produce worse output on another, so versioning them separately lets you ship half a change. Keep them in one object, with an explicit version, and make the object the only thing the request path reads.

{
  "config_version": "support-reply-2026-04-11.3",
  "baseline": {
    "model": "provider-a/general-model-pinned-snapshot",
    "prompt_file": "prompts/support_reply_v12.txt",
    "temperature": 0.2,
    "max_output_tokens": 600
  },
  "candidate": {
    "model": "provider-b/general-model-pinned-snapshot",
    "prompt_file": "prompts/support_reply_v12.txt",
    "temperature": 0.2,
    "max_output_tokens": 600
  },
  "canary": {
    "enabled": true,
    "share": 0.05,
    "sticky_key": "account_id"
  }
}

Four properties matter here. First, both entries carry a resolved model snapshot, not a floating family name. Second, the prompt file is referenced by path and version, so the same file feeds both arms during the test; the comparison isolates the model. Third, the canary share is data, not code, so you can raise it without a rebuild. Fourth, sticky_key means a given account keeps hitting the same arm for the whole session, which stops a single user from seeing two different writing styles in one conversation.

Keep the config behind the same review as your application code. A workflow file in the repository is a good home; GitHub workflow files must live in .github/workflows and use YAML, and they can be triggered on push or pull request per the workflow syntax reference. A pull request that bumps config_version and changes one model string is a one-line diff with a full test run attached as the quickstart for GitHub Actions shows.

One rule pays for itself repeatedly: resolve the model identifier to a dated snapshot or a specific revision at pin time, and store that resolved string. Aliases that point at "the current" model are convenient in prototypes and dangerous in production, because the provider can repoint them and your baseline silently becomes a different model mid-experiment.

Canary routing sends a defined percentage of requests to a new version and the rest to the previous one, which is the standard mechanism API gateways expose for staged releases as documented for Amazon API Gateway. Model serving platforms use the same idea with a traffic weight: you deploy the new model version, set a traffic percentage, and the remainder continues to the previous version until you promote it to full traffic per the Cloudera AI Inference canary documentation.

For a model swap you want this even more than for a code deploy, because the candidate's failure modes are behavioral. A model can return HTTP 200 with a confidently wrong answer, so a passing health check proves nothing. Structured feature flags solve this by returning a whole configuration object rather than a boolean, letting one evaluation choose the model, the prompt and the token cap together, with kill switches for instant reversion as MLflow's feature flag guidance explains. The same guidance notes that decoupling deploy from release means new code can ship in an off state and be enabled deliberately per Harness's overview of feature flags.

Choosing the canary share

Set the share from the sample size you need, not from a gut feeling. A rough target: you want at least a few hundred canary requests before you judge anything, and a few thousand before you trust a small difference in eval pass rate. The formula below gives the smallest share worth running when you know your traffic rate and how long you can afford to wait.

share = required_canary_requests / (requests_per_hour * hours_of_test)

At 20,000 requests per hour over a 12-hour window, a 5% canary yields 12,000 canary requests and 228,000 baseline requests. That is enough to detect a two-point drop in pass rate on a 500-example eval suite with room to spare. A 0.5% canary in the same window yields 1,200 canary requests, which is enough for a clear error-rate signal but too few to resolve a small quality difference. Start at 5% for a first canary on a high-traffic path, and accept that low-traffic internal tools may need a week rather than an afternoon.

Raise the share in steps only after the canary has cleared the thresholds below at the current step, and never raise it and change the prompt in the same commit.

Compare like with like. Every canary request should carry the config_version and the arm it was served by in its trace, so a metric filtered on arm=candidate is directly comparable to arm=baseline over the same window. Telemetry on latency, error rate, token consumption and eval scores is what turns a canary into an automated decision rather than a vibe check per MLflow.

Four signal groups are worth tracking, in priority order:

1. Hard failures. Non-2xx responses, timeouts, refusals and malformed structured output. These are unambiguous and should be watched with the tightest threshold.
2. Contract violations. If your output feeds a parser, count parse failures and schema validation failures separately from provider errors, because a model swap often breaks output shape before it breaks anything else.
3. Eval pass rate. Run a fixed eval suite against a sample of live canary and baseline traffic, or against a stored prompt set replayed on both arms daily. The fixed suite is the stable signal; live sampling catches inputs your suite has never seen.
4. Cost and latency per successful request. A candidate that is correct but twice as slow per response changes the product experience, and per-request token counts catch silent prompt-length inflation.

The comparison that matters is candidate versus baseline in the same window, not candidate versus last month. Seasonality, traffic mix and prompt inputs drift, and the pinned baseline is the only thing that holds the other variables still.

Write the thresholds into the canary config before the first request goes to the candidate, and let the pipeline enforce them. A rollback strategy configured ahead of time is the difference between an orderly revert and an argument at the wrong moment; Harness documents the pattern of declaring a failure strategy that triggers a pipeline rollback when specified error types occur per its rollback documentation. The same discipline applies here: the trigger condition is configuration, not judgment.

A workable default rule for a chat or generation endpoint:

• Hard rollback if the candidate's non-2xx plus timeout rate exceeds the baseline rate by more than 1 percentage point over any rolling 30-minute window with at least 200 requests.
• Hard rollback if contract violations exceed baseline by more than 2 percentage points.
• Hold and investigate if eval pass rate drops by more than 3 percentage points versus baseline on the same fixed suite, or if median latency rises by more than 40%.
• Promote only when all gates pass for the full planned window and a human has approved the config change that moves 100% of traffic.

Rollback itself is a one-line edit to the canary share, set to zero, plus a commit. That is faster and far safer than reverting an application deploy, which is why the rollback path should be the config flag and not the build. Modern rollback thinking stresses that a rollback is only trustworthy if it has been exercised; an untested rollback script is a hope, not a plan per Octopus's comparison of rollback strategies. Practice the flip in staging once per quarter.

There is a subtler technique worth borrowing from database practice: the expand and migrate shape. Add the candidate config alongside the baseline, migrate traffic gradually, then contract by deleting the losing config only after the winner has run at full traffic for a defined period as the expand and contract pattern is described by Tim Wellhausen. Keeping the old config around for a week costs nothing and means the revert is always available mirroring the guidance for zero-downtime database migrations. The same gradual, reversible structure is described for schema changes on the Prisma Data Guide in its walkthrough of expand and contract and in the expansion, migration and contraction steps outlined for backward compatible database changes in PlanetScale's writeup, and the reasoning transfers directly to model configs.

Suppose a support-reply endpoint serves 20,000 requests per hour. The baseline has a 1.5% hard-failure rate, a 0.8% contract-violation rate, a 94% eval pass rate on a 500-example suite and a median latency of 900 ms. The team sets a 5% canary for 12 hours, which routes 12,000 requests to the candidate and 228,000 to the baseline.

At hour four, the dashboard shows:

The contract-violation gate breaches. The candidate is wrapping JSON in prose on a few percent of calls, and downstream parsers are rejecting it. The on-call engineer sets the canary share to zero, commits, and the change is live in under two minutes because it was a config read, not a redeploy. The prompt is then tightened with an explicit output-format instruction, the eval suite gains twenty cases that reproduce the wrapping behavior, and a second canary runs a day later. That is the whole loop, and it took an afternoon rather than an incident review.

The insight the first search result usually misses: the gate that fires first is almost never the one you worried about. Teams set up a canary expecting quality regressions and get caught by output-shape changes, which is why contract violations deserve their own signal and their own threshold rather than being folded into "errors."

If you want to run this pattern across more than one provider without maintaining a separate client for each, Quanta Inference exposes text, image, video, audio and embedding models through one OpenAI-compatible API key, so the candidate arm of a canary can be a different vendor's model behind the same call site (https://quantaintelligence.ai/inference).

Once the first canary has run, most of the work is repetition. Store the eval suite version next to the config version so a pass-rate comparison can always be reproduced. Keep a short log of every model swap with its config version, canary dates and final decision, because the question "did we try this model on this prompt?" comes up often and the answer is cheap to keep. And set a periodic review of pinned models: snapshots eventually get deprecated, and a deprecation notice is a scheduled migration, not an emergency, if you have been tracking what you pinned.

The organizational payoff is that swapping models stops being an event. Product can ask for a quality improvement, engineering can propose a candidate, and the canary answers the question with traffic and numbers. The threshold, not the loudest voice in the room, decides.