Sizing the Review Sample for a Changed Process

A review unit is one comparable piece of completed work that a reviewer can pass or fail against a written rule. Define it before counting anything, and define it per process step, because a single unit spanning three steps gives you one ambiguous verdict instead of three usable ones.

Start with inputs the charter already asks for: the current workflow's trigger, steps, people, systems and completed outcome, plus a baseline with a source and a window the controlled POC charter. Turn that into a table with one row per step the revision touched.

Two rules keep this table honest. First, write "unknown, discovery required" where evidence does not exist rather than inventing precision, which is the same instruction given for charter baselines the controlled POC charter. Second, keep the pass rule written down before review starts. A pre-registered plan is what separates planned analysis from unplanned analysis, and the same discipline applies to a review protocol even though a pilot is not a journal study the preregistration overview.

You need a starting number for each step, because the sample size depends heavily on how rare the error is. The intuition is the same as in a survey: the nearer an event is to 50 percent, the more observations you need to pin it down, and rare events need far more the survey sample size calculator.

Three ways to get the baseline, in descending order of quality:

1. Pull a historical sample from the same queue and have a reviewer score it against the pass and fail rules. Fifty to one hundred items per step is a workable first cut for a common error.
2. Use existing audit or rework logs, provided the reason codes match your rules. If they do not, treat the number as a rough prior only.
3. If nothing exists, run a one-week discovery pass and label the result explicitly as provisional.

Worked mini example. A routing step shows 6 errors in 100 sampled items, so the working baseline is 6 percent. The pilot owner wants to catch a doubling to 12 percent. That is a different sizing problem from catching a rise from 6 to 8 percent, and the second will need several times the review volume.

Do not forget the model's own measured accuracy is not the baseline. Model benchmarks tell you about the model under the vendor's test conditions, not about your step with your data and your handlers. Charter guidance is explicit that a prepared demo is not pilot evidence, and that a pilot uses an agreed evaluation unit, representative cases, explicit thresholds and a record of failures the controlled POC charter.

This is the decision the whole calculation hangs on, and it is a business decision, not a statistical one. Ask: if the error rate at this step rose by this much, would I stop the pilot, change the process or accept the drift? The smallest rise you would act on is your target shift.

Two guardrails make the choice defensible. First, set the threshold before data collection: the four-metric pre-launch framework requires a primary outcome metric, a measured baseline, a minimum viable target and guardrail metrics, all agreed before the pilot begins, because changing them afterwards invalidates the decision the pre-launch success criteria framework. Second, note that the sample spans multiple steps, so each step's shift needs its own minimum. A doubling at the approval step may be a stop condition, while a two-point rise at summarization may only warrant a queue note.

There is one more discipline worth borrowing from research design: a pilot used to test a hypothesis you will later act on cannot also be the study that generates the hypothesis from the same data. The same data cannot both generate and test a hypothesis without inflating false positives the preregistration overview. In practice, run the discovery week to learn the baseline, then lock the thresholds and run the measurement review.

Power analysis answers a simple question: how many observations do we need so that a real shift of a stated size has a stated probability of showing up as a signal? Power depends on the baseline rate, the size of the shift, and your tolerance for a false alarm, and lower baselines or smaller shifts both drive sample size up the sample size calculator explainer. If you enroll too few units you risk a false negative, and the review costs get wasted either way a power analysis primer.

The arithmetic for a shift in a proportion, at 95 percent confidence and 80 percent power, is:

n per comparison ≈ ( z(1-alpha/2) * sqrt(2*p*(1-p)) + z(power) * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 / (p2 - p1)^2

where p  = baseline error rate at the step (for example 0.06)
      p1 = p, p2 = the shifted rate you must catch (for example 0.12)
      z(1-alpha/2) = 1.96, z(power) = 0.84

Worked example with those numbers: the pooled variance term uses p = 0.06, giving sqrt(2 x 0.06 x 0.94) = 0.336; the shift term gives sqrt(0.06 x 0.94 + 0.12 x 0.88) = 0.383; so n is approximately (1.96 x 0.336 + 0.84 x 0.383) squared, divided by 0.06 squared, which lands near 300 reviewed items per comparison at that step. Notice what happened: the second term, the variance of the shifted rate, dominates the first, because a doubled error rate carries more variance than the current one. That asymmetry is why experienced pilots size against the shifted rate, not the baseline alone.

Now hold the confidence and power fixed and change only the smallest shift worth catching, using the same formula shape with p = 0.06:

The exact numbers depend on the test you choose (a two-proportion comparison, a sequential check, an equivalence test), so treat this table as a planning shape, not a promise. Two general relationships hold regardless: halving the margin approximately quadruples the requirement in proportion-based planning, and raising confidence from 95 to 99 percent raises the requirement further the survey sample size calculator. Guidance on feasibility pilots makes a related point: pilot sample sizes are often justified by their purpose rather than by a formal power calculation, and should be reported as such the feasibility pilot guidelines. If your target shift is so small that the required sample dwarfs your queue, the honest move is to widen the target or narrow the step.

One more refinement. Feasibility studies often cover several aims at once, so the review sample needs to satisfy the most demanding aim, not the average one the feasibility pilot guidelines. In a process-changing revision, the most demanding step is usually the one with the lowest baseline rate and the highest consequence.

Now the part most sizing exercises skip: the schedule. Take the required reviewed items per step, multiply by the minutes each review takes, and compare with the hours you can actually staff.

Worked example. The routing step needs about 300 reviews in a four-week measurement window, and each review takes 9 minutes including the reason code. That is 2,700 minutes, or 45 hours across four weeks, or roughly 11.25 hours per week. If two reviewers each have 6 hours a week available and the approval step independently needs 8 hours a week, total demand is about 19.25 hours against 12 hours of supply. The pilot is oversubscribed on paper before a single item arrives.

That gap has exactly three fixes, and only three. Shrink the scope: review one step instead of three for the first two weeks. Raise the target shift: catch a 6 to 9 point move instead of 6 to 7. Increase capacity: add a reviewer or reduce minutes per review by templating the pass and fail rules. If none of those is possible, the correct answer is not a smaller sample with a bigger error bar. It is a stop, because the review you planned cannot be delivered and an unreviewed process change in production is exactly the risk the charter's human control section exists to prevent the controlled POC charter.

Track this weekly. The measure to watch is required sample minus delivered reviews, not review backlog in the abstract, since a backlog can look stable while the required sample grows.

Stop conditions are the underrated half of any loop. Resource limits, operating boundaries, escalation triggers and failure behavior should all decide when a run ends, whether the loop is an agent cycle or a pilot program the agent loop architecture. The same page notes that autonomy and safety come from the surrounding system deciding what is actually allowed to happen, not from the loop running longer. Review capacity is that surrounding system for a pilot.

A stop condition should be written in the charter, with a named owner and a decision date. The charter template's scale, revise, stop section exists for this, and the same document warns that a project without a decision owner can keep running after its learning value ends the controlled POC charter. Rumelt's rule applies here in plain terms: good strategy names the obstacle, and reviewer hours are an obstacle you can name in advance.

Here is a working decision rule that has been useful in practice:

• Compute the weekly required review hours for each step.
• Compare with committed reviewer hours, not aspirational hours.
• If required hours exceed committed hours for one week, flag it and reduce intake.
• If required hours exceed committed hours for two consecutive weeks, stop intake, run a short review, and fix the process or the queue before resuming.

Treat any threshold breach as a genuine stop, not a warning, and write it into the charter with a named recipient for the escalation the charter scope and failure handling sections. When you do stop, run a short, blameless review focused on what the review queue revealed: what the process changed, which step consumed the hours, and what would have to be true to restart. The postmortem literature is clear that the value of these reviews comes from institutionalizing improvement rather than assigning fault, and that action items must be tracked or the exercise is worthless the incident postmortem guide. AI-specific guidance makes the same point with a bias toward prevention over blame the AI incident post-mortem framework.

A short review record that works:

Stop date:
Steps that exceeded capacity:
Required hours vs committed hours, by week:
First week the gap appeared:
Cause: process step, queue design, review rule ambiguity, staffing
Change made before restart:
Restart condition and owner:

Two of those fields do most of the work. "First week the gap appeared" stops a late discovery from being blamed on the last week, and "change made before restart" keeps a stop from becoming a pause with no fix.

If you are running a revised pilot across several steps and the review arithmetic is getting harder to maintain by hand, an expert working inside your context can keep the step map, baselines and weekly capacity check in one place. Quanta Experts (https://quantaintelligence.ai/agents) is built for that kind of recurring analytical work and hands back finished deliverables. For teams that need a decision on whether the process change is worth the review burden at all, a free discovery call with Quanta Consulting (https://quantaintelligence.ai/calendar) is the fastest way to get a tailored recommendation.