How to Run an AI Pilot That Ends in a Decision

Put the plan on one or two pages and get it agreed before anyone starts. Evan Rose of ROSE argues for a two-page pilot charter that forces decisions on scope, objectives, governance and metrics before launch, and leaves an auditable record of what was agreed (Medium, Rose Digital). The Institute of AI PM proposes a similar seven-part template: objective, scope, participants, success criteria, timeline, risk plan and evaluation method (Institute of AI PM).

A charter sized for a small company:

Pilot: AI drafting of replies to quote requests
Owner: Office manager
Decision date: [fixed date, about six weeks out]

Question: Does AI drafting cut the time to send a quote without
more errors or lost jobs?

Scope: Quote requests by email for residential jobs only.
Out of scope: commercial jobs, phone requests, follow-ups.

Baseline (last 4 weeks): median time to send a quote, error rate
found at review, share of quotes accepted.

Go if: time to send drops by the agreed amount
Guardrails: error rate and acceptance rate do not get worse
Stop if: any quote goes out with a wrong price, or staff stop using it

Data rules: no customer payment details in the tool.
Who can stop it: owner or business owner, at any time.
If it works: who runs it, what it costs per month, what changes.

The objective has to be falsifiable. The Institute of AI PM gives a useful test: if any result could be spun as a success, rewrite the objective (Institute of AI PM). It also offers a scoping sentence worth copying: "At the end of this pilot, we will know whether [capability] can [measurable improvement] for [user group] within [time frame]." If you cannot fill every bracket with something specific, the scope is too broad. You cannot pilot "AI for the office". You can pilot "AI drafting for residential quote emails".

The most important line in the charter is the decision rule. Spark Academy lists setting success criteria afterwards as a classic pitfall, because then any outcome can be justified, and that is exactly how teams keep piloting for months (Spark Academy).

The Institute of AI PM recommends three kinds of criteria (Institute of AI PM):

1. A primary metric with a threshold that triggers a go decision.
2. Guardrail metrics that must not get worse beyond an agreed tolerance, such as customer satisfaction or error rate.
3. Qualitative signals, such as staff feedback, that you collect but that do not override the numbers.

Rose's charter adds explicit exit criteria: the thresholds at which you pause or stop, not just the ones at which you continue (Medium, Rose Digital).

Agree who decides. The Institute of AI PM's analysis of why pilots fail warns that if success metrics can be contested after the fact by someone who never agreed to them, the pilot fails even when the numbers are good. Its advice is to get written agreement on what counts as a win from the person who controls the budget, before day one (Institute of AI PM). In a small company that is usually the owner, and a two-line email is enough.

Who uses the tool during the pilot shapes the result. The Institute of AI PM points out that picking only enthusiasts inflates adoption numbers, while picking only skeptics guarantees poor engagement, and neither predicts what a broad rollout will look like (Institute of AI PM). Its template recommends selecting participants rather than taking volunteers, and keeping a comparison group doing the same work without the tool.

A small company rarely has enough people for a formal control group. Workable substitutes:

• Before and after, using the baseline from the charter, when nothing else changes in the period.
• Alternating, with the tool on for some days or some request types and off for others.
• Side by side, where one location or shift uses the tool and another does not.

Check that the volume is enough. The same template warns that a process that happens only a handful of times a week may not produce enough data points in a short pilot to tell a real effect from noise (Institute of AI PM). If volume is low, extend the window or pick a busier process.

Change one thing. If you introduce the AI tool, a new process and a new team structure at once, you will not know which one caused the result. Keep everything else constant for the duration.

Demos prove very little. Spark Academy lists testing on demo material as a pitfall and notes that the excitement of a demo tells you nothing about an ordinary Tuesday morning (Spark Academy). Run the pilot on real requests from real customers, handled by the people who will use the tool afterwards.

Real work also exposes the production problems early. Rubicon's review of why pilots stall lists the usual causes: data that only works in the demo because the proof of concept used an exported sample, security and compliance reviews that arrive after the system is built, no agreed way to evaluate quality, unknown running costs at production volume, and no business unit willing to own and fund the result (Rubicon). Its advice is to use production data sources from the start, even if the pilot reads only a subset.

MindStudio frames the gap as engineering: a demo skips authentication, access control, monitoring, deployment and maintenance, so a pilot can look nearly finished while most of the work remains (MindStudio). Organizations that escape pilot purgatory treat production readiness as a starting requirement, not a later phase.

For a small company using a bought tool, the production questions are simpler but still need answers before the pilot starts:

• Who will own the tool and its settings after the pilot?
• What will it cost per month at full volume, not pilot volume?
• Which data is allowed in it, and has someone read the vendor's data terms?
• Who trains the rest of the team, and when?

The Institute of AI PM's failure analysis notes that many pilots are approved with no plan for what happens if they succeed (Institute of AI PM). Answer these questions in the charter so a "go" can happen the next day.

The Institute of AI PM suggests three phases: setup, covering configuration, onboarding and baseline measurement; execution with weekly check-ins; and evaluation, ending in a scheduled decision meeting. It argues that a pilot for a non-infrastructure AI feature that runs much longer than about two months usually signals scope creep or unclear criteria (Institute of AI PM).

A weekly check-in can take fifteen minutes:

1. Read the primary metric and the guardrails.
2. Review a sample of the AI's output, including any errors.
3. Note what participants say is working and what is not.
4. Decide whether any stop condition has been triggered.

Name who can stop the pilot. Rose's charter includes guardrails for data, security, privacy and safety, and names who has authority to halt the pilot on any breach (Medium, Rose Digital). The Institute of AI PM's risk plan adds watching for participants gaming the metrics, by measuring downstream outcomes rather than tool usage alone (Institute of AI PM).

Resist changing the rules mid-pilot. Fixing an obvious configuration error is fine. Changing the success criteria because early numbers disappoint defeats the purpose. If you must change the method, write down what changed and why.

Book the decision meeting on the day you start. Rubicon describes each stage gate as a meeting with the business owner, IT, security and the delivery team, where the options are go, change or stop, and where stopping means the process worked (Rubicon). In a small company, that meeting may be the owner and the pilot lead, but it still needs a date and a written result.

Spark Academy's outcome table is a good agenda (Spark Academy):

Write the decision down in a few lines: the result against each threshold, the decision, the reasons, and what happens next. A stopped pilot with a clear record is valuable. It keeps the next person from repeating the same experiment, and it shows which conditions would need to change before trying again.

Scaling deserves its own care. Spark Academy notes that a pilot runs on the people closely involved in it, while production runs on agreements: an owner who maintains it after the builder moves on, proper access for each team, rules about which data can go in, and checks built into the workflow (Spark Academy). Treat each new team as a smaller pilot with its own criteria.