Choosing a Baseline to Compare an AI Pilot Against

The cleanest baseline is a fixed window immediately before the pilot, using the same system of record the pilot will use. This example is illustrative, with round numbers chosen to make the arithmetic visible.

A support team handles Tier 1 billing tickets. The pilot will add AI drafting to 20 agents. Before launch, the team records six weeks of the current process, broken into weekly buckets so the spread is visible:

The center of that baseline is a median handle time of about 14.3 minutes, with weekly values spanning 13.8 to 15.1. That spread is the point. A pilot that lands at 14.0 minutes has not clearly improved anything, because 14.0 sits inside normal week-to-week variation. A pilot that lands at 11.5 minutes is outside the range the team has ever recorded in a normal week, which is a much stronger signal than a single before-and-after pair.

Compare that with the alternative. A team that "remembers" 14 minutes, runs a four-week pilot that lands at 13.2, and declares success has actually measured nothing except that some weeks are faster than others. The recorded baseline tells you whether your result is bigger than the noise.

Two practical additions make the comparison group explicit:

• Pick the comparison group before the pilot. Either a matched control group doing the same work without the tool, or a historical comparison against the recorded baseline period. The pilot template above recommends an equal-sized control group of 15 to 30 participants for most pilots (Institute of AI PM) and warns that volunteer-only selection inflates adoption numbers.
• Log the same fields in both periods. If the baseline captures escalation rate but the pilot dashboard does not, you cannot compare the two, so the guardrail is invisible at decision time.

A stage-gate view helps here too. In the pilot stage, the exit criterion is that the business metric improved against baseline, alongside a passed security review and a confirmed owner (Rubicon). The baseline is not paperwork; it is the gate condition.

Log every metric you will judge the pilot on, plus the guardrails you will not let regress. A useful split, drawn from pilot design guidance, is a primary metric with a threshold that triggers a go decision, guardrail metrics that must not fall beyond a defined tolerance, and qualitative signals that inform but do not override the numbers (Institute of AI PM).

The distinction between technical and business metrics matters more than it sounds. A model's accuracy score is not a business metric; the business metric is the outcome the AI is supposed to move, like claims processed per day or time to resolution (Institute of AI PM, why pilots fail). Logging accuracy in the baseline is impossible if the process is human, so the baseline naturally forces you toward outcome metrics, which is what the decision needs anyway.

One subtle trap: measure downstream outcomes, not just tool usage, because participants will game usage metrics if that is what you watch. The pilot template lists this explicitly among common risks (Institute of AI PM).

Where logging is concerned, start from day one of the baseline and keep the same fields during the pilot. Retrofitting logging after launch creates gaps that undermine attribution of results (Agility at Scale). A minimal event log for a support-style pilot looks like this:

{
  "case_id": "TCK-104882",
  "week": 3,
  "channel": "email",
  "category": "billing",
  "agent_id": "A-217",
  "tool_used": false,
  "first_response_min": 22.5,
  "handle_time_min": 15.8,
  "reopened_within_7d": false,
  "escalated": false,
  "csat": 4
}

During the baseline every record has tool_used: false. During the pilot the field varies, which is exactly how you separate the treatment from the comparison.

The comparison group is the population you compare the pilot against, and it needs the same care as the treatment group. There are three workable designs, and they fail in different ways.

Guidance on pilot design is direct about selection: pick participants randomly from the eligible pool rather than from volunteers, and plan for a control group of comparable size (Institute of AI PM). Volunteer-only pilots tell you what enthusiasts can do, not what a broad rollout will look like.

The comparison group needs the same baseline as the treatment group. If your control team was never measured during the pre-pilot period, you cannot tell whether the pilot team beat them or whether the control team simply had a quiet quarter. Record both populations in the same window, with the same definitions.

Define eligibility explicitly, in writing. "English-language Tier 1 support tickets in the billing category, handled by agents in the US East team, for six weeks starting on a fixed date" is the level of specificity pilot scoping guidance recommends (Institute of AI PM). Everyone else is out of scope, and that includes the eager volunteer from another region who wants in.

For case-level pilots (document processing, claims triage, code review), the comparison group is best defined at the case level: a random share of incoming work runs through the new process and the rest runs through the current one. That gives you a contemporaneous comparison, which neutralizes seasonality almost completely, and it lets the sample accumulate faster than splitting people.

Three forces make a naive before-and-after comparison misleading.

Seasonality. Support volume, error rates and processing times move with the calendar, promotions, holidays and billing cycles. A four-week pilot in a quiet month compared to a busy baseline will look good for reasons that have nothing to do with the model. The fix is either to measure the baseline across at least one full cycle of whatever recurs in your business, or to use a contemporaneous comparison group so that both arms experience the same conditions. The agency pilot run by state insurance regulators is a decent model of the second approach: the participating states use the same tool across different work types and lines of business, and they coordinate meetings specifically to share what they learn rather than each drawing conclusions in isolation (NAIC).

Learning effects. People get better at a new tool over the first weeks, and they also get worse at the old process if the pilot distracts them or if their colleagues are being measured. A short pilot captures the learning curve, not the steady state. Practical responses: exclude the first week or two from the pilot comparison and say so in the analysis plan in advance, or run the pilot long enough that the learning curve is a small share of the window. The pilot template phases setup (including baseline measurement), execution and evaluation, which gives you a place to carve out a warm-up period (Institute of AI PM). For office applications, running a few weeks to a quarter is typical, and what matters more than the length is that the end date is fixed at the start (Spark Academy).

Small samples. A team of 12 agents handling 40 tickets a week generates 480 cases in a ten-week window, which sounds large but supports only coarse conclusions about big shifts. The guidance on minimum viable sample sizes exists precisely because significance is a design constraint, not an afterthought: the pilot template requires a stated minimum sample size and a pre-registered analysis plan, including how you will handle missing data and who conducts the analysis (Institute of AI PM). A pragmatic decision rule for practitioners:

• If the observed change is smaller than the spread you saw across baseline weeks, treat it as unresolved, not as a win.
• If the change is large relative to that spread and consistent across weeks, you can act on it even with a modest sample.
• If you cannot reach a verdict within the pre-agreed window, the honest outcome is "insufficient evidence within the pilot," which is a valid gate result and should be recorded as such (Rubicon).

One more constraint worth respecting: human performance measures need to be assessed for reliability, validity and sensitivity, and a review of 42 studies in high-fidelity settings found only a few measures met acceptable levels on all three (PubMed). Translate that to your pilot: prefer a small number of metrics you actually trust and can define identically across both periods over a dashboard of twenty that everybody interprets differently.

The comparison is only defensible if the numbers and the rules were fixed before the pilot started. A two-page charter that captures scope, objectives with baselines and targets, success and exit criteria, and guardrails gives you that record and an audit trail of decisions (ROSE). Note the worked example in that charter format: reduce average handle time from 18 to 12 minutes, with baseline and target stated side by side.

Freeze the baseline as a versioned artifact, not a slide. A minimal record:

• The definition of each metric, with the query or source system named.
• The window covered, and any weeks excluded, with the reason.
• The comparison group definition and how members were chosen.
• The target and guardrail thresholds, plus the decision date.
• The named analysis owner and the statistical test you will use.
• A signature or explicit sign-off from the business owner and the pilot owner.

Then commit to a rule about changing anything. Pilots are allowed to change scope in the middle, but the change has to be documented and justified, and mid-pilot course changes should be recorded, not quietly absorbed into the results (Institute of AI PM). Without that discipline, a pilot that runs past its window with shifting goals is just an application that went into production without anyone deciding (Spark Academy).

Escape routes are the enemy here. An organization that has several pilots running should compare them on the same terms, with owners and success criteria, and decide stop, continue or scale up for each one at a set date (Spark Academy). A single baseline discipline applied across pilots does more for credibility than any individual pilot result.

If your team wants a structured way to stand up the measurement layer itself, this is the kind of work Quanta Experts handles: each expert reads your context and hands back finished deliverables, including a pilot charter and baseline log you can sign. You can see how that works at Quanta Experts.

Run through this before day one, and again at the decision meeting.

• The primary metric is defined identically for baseline and pilot, with the source system named.
• A fixed baseline window is recorded, long enough to cover the normal weekly and seasonal spread.
• The comparison group is explicit, with eligibility rules and how members were selected.
• Guardrail metrics (quality, escalation, customer and staff experience) are logged in both periods.
• Sample size and analysis method are written down, including how missing data is handled.
• Learning-curve exclusions and seasonality handling are stated before the pilot begins.
• Target and guardrail thresholds are agreed, dated and signed by the business owner.
• A decision meeting date exists, with go, adjust and stop as named outcomes.
• Any mid-pilot change to scope or metrics is documented with a reason.

If half of these are missing, the comparison at decision time will not survive a skeptical finance partner. If most are present, you can defend a result that is unflattering as easily as a favorable one, and that is the point.

Be honest about the limits. A recorded baseline cannot tell you what would have happened without the tool if the market moved; only a randomized comparison can approximate that. Nor is the baseline period frozen forever: if the process changes during the pilot for unrelated reasons, say so and note that attribution weakens. What the baseline buys you is a reference point that everyone agreed to before the results were known, which is the single most valuable thing a pilot can have.