Widening an AI Feature from 5 Percent to Everyone

The first cohort is not a random sample of everyone. It is a random sample of the safe part of everyone. Do this in two steps.

Step one, exclusion. Before you select anyone, list the segments where a wrong answer is expensive or regulated: accounts in collections or dispute, anyone flagged as vulnerable, enterprise contracts with named account managers, and any jurisdiction where your legal team has not signed off. Exclude those first. Microsoft makes the same point about staged rollout for authentication changes: keep the tested population to pilot groups, maintain a working fallback, and do not put everyone in the new path until you have a transition plan (Microsoft's staged rollout guidance).

Step two, selection. From the remaining population, assign users to treatment with a deterministic hash, not a hand-picked list.

import hashlib

def in_cohort(user_id: str, stage: str, percent: int) -> bool:
    digest = hashlib.sha256(f"support-agent:{stage}:{user_id}".encode()).digest()
    bucket = int.from_bytes(digest[:4], "big") % 100
    return bucket < percent

Including the stage name in the hash input gives you two properties. The distribution is sticky within a stage, so a user who sees the agent keeps seeing it across sessions, which removes the confusing experience of the agent appearing and disappearing. And when you move from 5 to 15 percent, the new 10 percent is a fresh draw rather than "the same 5 percent plus more", which lets you compare the newly exposed group against the group that has already survived a stage. Percentage-based flags work this way in general: a stable hash of a user identifier decides treatment, and changing the percentage shifts the boundary rather than reshuffling everyone (percentage rollout flags explained).

One operational detail worth settling early: 5 percent of a small support volume may be fewer than 500 conversations in a week. If so, either widen to the volume you need or extend the hold period. Do not lower the sample requirement, because a gate evaluated on 80 conversations will flap.

Feature-flag discipline matters more for AI than for normal software because AI failures are probabilistic. A prompt or model change can pass a thousand inputs and fail on the thousand-and-first, and many of those changes live in configuration rather than code, so your CI pipeline never sees them (feature flagging for AI models).

That argues for putting every AI-relevant parameter behind one flag payload: model identifier, prompt template, temperature, retrieval configuration, tool permissions, and the exposure percentage itself. Then a rollback is one flag change, not a deploy.

The second layer is degraded mode. When the agent is slow, uncertain or returning blocked content, it should not simply keep going. Continuity practice for AI systems separates detection from mitigation: detect the issue, disable the risky capability (autonomous tool execution, sensitive generation), keep the safe functions (read-only lookup, template answers), tell the user that capability is reduced, and log what happened (enterprise AI continuity, fallbacks and failover). Your handoff to a human is that degraded mode. Test it under load, because an agent that fails into a queue nobody is staffing is worse than one that fails closed.

The third layer is the rollback rehearsal. A rollback plan that has never been executed is a document, not a capability. Force a rollback in staging, corrupt a retrieval index, kill the primary model provider, and time the recovery. General rollback guidance makes the same point: the procedure gains credibility only through deliberate failure testing, and the useful outputs are how long it takes, what the dependencies are, and which manual steps can be automated (planning a rollback strategy).

Here is a support agent over a 100,000-conversation-per-month queue, with the human baseline from the table above.

Stage 1 at 5 percent of 100,000 is 5,000 conversations in a month, but support volume is not evenly spread. If half the month's volume lands in the first week and your weekend traffic is 20 percent of weekday, a 7 day hold might give you 1,100 conversations, comfortably over 500. If your volume is 20,000 per month instead, 5 percent is 333 conversations a week, so hold stage 1 for two weeks. The rule is the volume floor, with the day count as a minimum rather than the driver.

The stage 3 gate is the one teams forget. At 40 percent exposure, roughly 40 percent of your queue is being offered to the agent first. If containment is 60 percent, human load drops by roughly 24 percent of total volume, which sounds like a benefit until you notice that the human queue is now skewed toward the hardest conversations. Handle time per ticket rises, SLA can slip, and the humans get a worse job. Measure queue SLA and average handle time on the human side at every stage, and treat a breach as a gate failure even if every agent metric is green.

Also watch for the compounding effect on cost. A 100,000-conversation month at 5 percent is 5,000 agent conversations, a negligible bill. At 100 percent it is the whole queue. Running cost at full exposure was one of the reasons analysts cite for stalled pilots: nobody estimated token usage, hosting and support at production volume (why pilots stall). Do that arithmetic before stage 3, not after.

There are four clocks, and the longest one governs.

The first is the calendar clock: seven days minimum for stages 1 and 2, because you need to see a full weekly cycle. Customer behavior on Monday morning is not customer behavior on Saturday night, and a bot that handles weekday password resets fine may face weekend edge cases you have never seen.

The second is the volume clock: 500 conversations at stage 1, 1,500 at stage 2, 4,000 at stage 3. Small samples produce wide confidence intervals and gates that flicker. If you must choose between the two clocks, obey volume.

The third is the novelty clock. Early-stage users know they are talking to something new and behave differently, which inflates or depresses your metrics. A hold of two weeks at stage 1 gives you a second week of data you can compare against the first. If the two weeks diverge materially on quality but not on latency, you have a novelty effect, not a regression, and you should keep watching rather than advancing.

The fourth is the human-recovery clock. Your support team's capacity to absorb a rollback shrinks as exposure grows. At 5 percent, reverting to humans costs a rounding error in staffing. At 100 percent, a rollback means absorbing the entire queue at once, which may not be possible on a Tuesday afternoon. Before stage 4, confirm that a rollback to, say, 40 percent is survivable, and set the automated guard to cut to that level rather than to zero.

A guard is a gate that fires continuously rather than at a stage boundary. Decide the action for each one in advance, and write it down with a name attached. Vague escalation is how incidents become meetings.

A workable set:

Two design notes. The pause and the cut are different actions: a policy violation should stop widening immediately while you investigate, but it should not necessarily deprive current users of the agent if the incident is contained. And every automated action needs a named human who is paged, because an automatic cut that nobody notices will quietly sit at 40 percent for three weeks.

Provider failover deserves its own paragraph. Switching models is not just switching an endpoint. The substitute needs to be on a list that governance and legal have already approved, with data residency, DLP and guardrail checks satisfied (fallback and failover design). That approval takes weeks, so start it during stage 1. If you are routing through a single API that fronts multiple models, the approved-list constraint still applies; it just lives in configuration rather than in your code.

A final guard is the knowledge fallback. Automation quietly erodes the human ability to do the work it replaced: oversight lightens, headcount shifts, and the people who remain have never run the process manually (when the AI goes down and the experts are gone). Practically, that means keeping a runbook that names who takes over, and keeping at least one team member per shift who has handled the queue without the agent in the last 90 days.

Stages 1 and 2 are small enough that a competent engineer can babysit them. Stage 3 is where the operation has to stand on its own. Four artifacts should exist before you widen past 15 percent: a golden evaluation set built from real production conversations (not curated ones) that runs on every prompt or model change, a dashboard that shows all six guards with their thresholds drawn on it, a written rollback procedure with a rehearsal date and a measured recovery time, and a cost model that projects the bill at 100 percent exposure.

The evaluation set is the one people skip, and it is the one that matters most. Pilots succeed partly because humans quietly patch the gaps, and production removes that patch (why generative AI pilots never reach production). A golden set drawn from the conversations your agent actually contained in stage 1 and 2 is your best proxy for how it will behave when nobody is watching. Build it while the sample is small and the gaps are visible.

If you would rather not assemble the model routing, failover configuration and per-model cost tracking yourself, Quanta Inference puts text, image, audio and embedding models behind one OpenAI-compatible key with prepaid US dollar pricing, which makes the failover list and the cost model simpler to operate: Quanta Inference.