Reading a Vendor's Time-and-Motion Study: What to Check

Ask for the participant count and, separately, the repetition count. A study of three people doing two runs each gives you six observations, not six people's worth of evidence. Task-time data are skewed, with a long tail of slow cases, so a handful of runs produces an average that swings depending on whether one slow case got included.

There is a standard way to size this. MeasuringU's guide to sample size for task times applies the continuous-data approach, which uses the standard deviation of the pilot times, the desired margin of error and the t-distribution to estimate how many runs are needed MeasuringU on calculating sample size for task times. The practical implication: if pilot times varied by a factor of three between fastest and slowest, you need far more repetitions than a vendor demo typically uses.

Worked example with invented numbers, chosen to show the arithmetic. Suppose a pilot of six runs on one task gives 4, 5, 5, 6, 8 and 14 minutes. The mean is 7.0 and the standard deviation is about 3.7 minutes. If you want a 95 percent interval half-width of 1 minute, the naive calculation returns roughly 53 runs. Drop the slow case and the standard deviation falls to about 1.1 minutes, and the requirement falls to about 6 runs. That is the whole game: the answer depends entirely on whether the slow case is real. A study that quietly excludes slow cases, or runs each task twice, can produce whatever number the vendor wants.

Ask how participants were chosen. If the vendor recruited volunteers from an internal power-user group, or from customers who had already succeeded with the product, the sample is not representative of your staff. Sampling error is the difference between an estimate from a sample and the true population value, and it shrinks only with a well-designed, adequately sized sample; non-sampling error such as refusals and inaccurate answers is harder to quantify and would exist even in a full census the ONS explanation of uncertainty in surveys. That is exactly why self-reported times deserve extra suspicion.

For work sampling rather than whole-task timing, the sizing logic is binomial, and the choice of interval method matters. Disney's review of activity sampling concludes that the widely used Wald interval performs poorly when the observed proportion is near zero or one, and recommends the Clopper-Pearson interval instead Disney, Revisiting activity sampling. If a vendor reports percentages of time spent on categories and does not state the observation count and the interval method, treat those percentages as descriptive rather than inferential.

This is where most vendor studies fail. Usability research identified task-selection bias long ago: evaluators gravitate toward tasks the product can already perform, which biases both the results and the participants Cordes, Task-Selection Bias. Cordes recommends mixing user-defined tasks with product-supported tasks, and notes that participants warned the system may not support every function give up sooner than those who are not warned. Either way, the task set shapes the result before a single stopwatch starts.

Your countermeasure is a frequency-weighted task list built from your own data. Pull the last quarter of tickets, requests or transactions, count how often each task type occurs and sort by volume. Then check the vendor's list against it. A study that times your eight most common task types is informative even if it is small. A study that times ten tasks you have never seen in production is a demo.

Watch for these patterns:

• Tasks the product was designed around. The agent handles the structured request and a human handles the messy follow-up, but only the structured part is timed.
• Composite tasks. "Resolve a customer issue" is timed as one unit even though it contains a lookup, a judgement and a written reply with different automation profiles.
• Easy-task drift. People under load gravitate toward quick, easily completed items, which lifts short-term throughput at the cost of longer-run performance the Harvard Business School working paper on task selection and workload. A benchmark that lets participants choose from a menu can reproduce that effect without meaning to.
• A frozen task set. If the vendor's list never changed after the pilot began, ask why nothing was added.

Ask how task completion was defined. The gap between activity and outcome is a recurring source of inflated claims: an agent that completes 5,000 tasks has told you nothing about whether those completions resolved anything, and the distance between completion rate and outcome rate is where ROI cases break the Pickaxe guide to AI agent ROI metrics. In a time study the equivalent question is when the timer stopped: the moment the agent produced output, the moment a human accepted it, or the moment the downstream system confirmed the record was correct. Only the third is a defensible endpoint.

Rank the methods before you read the numbers.

Continuous observation with a stopwatch or screen recording is the strongest. Observers record start and end times for each activity as it happens. The cost is observer time, and the risk is distortion from being watched, which is why good studies rotate observers and run a settling-in period.

Work sampling records what is happening at random instants. It is cheaper and less intrusive but less precise for individual durations. One comparison used real time-and-motion data from medical residents to simulate work-sampling observations and found that work-sampling results differed by 20 percent or more of the estimated value for eight of ten activity categories the PubMed record of the comparison study. Keep that number in mind when a vendor quotes a work-sampling figure to one decimal place.

Self-report is the weakest method and the most common in vendor decks. It is easy to inflate in both directions: people overestimate how long tedious tasks take and underestimate how long they spend inside the product. On top of self-report sits observational reactivity. Time studies are historically controversial because being timed changes behavior, and unions in particular have treated them with suspicion BSM Lean on work measurement and standards. A vendor study that never mentions the observer effect has not thought about it.

A practical hierarchy:

1. Same tasks, same participants, timed by a trained observer before and after, with a written completion rule. Strongest.
2. Same tasks, matched groups, timed by observation. Strong if the groups are genuinely matched.
3. Observed treatment group, baseline reconstructed from system logs. Acceptable if the logs measure the same thing.
4. Observed treatment group, baseline estimated by asking participants to remember. Weak.
5. Treatment group only, all times self-reported. Not evidence.

Treat each of the following as a hard stop. One is a question, two is a rejection, three or more is a demo dressed as research.

Two more deserve separate mention.

Savings that never land anywhere. Even a well-measured time saving is a cost saving only if the freed hours go somewhere identifiable, and most organizations absorb them into meetings and low-value work the Pickaxe ROI guide. Before you sign, decide where the hours will go and how you will verify they arrived.

No cost side at all. A time study measures task time, not total cost. Rework, scrap, warranty claims and complaint handling belong in the cost of poor quality, and leaving them out is a standard way to overstate a benefit SixSigma.us on COPQ. If the study shows time saved but says nothing about the error rate before and after, you have half a business case.

Suppose a vendor reports a 19 percent time saving on a task your team performs 180 times a day. Their appendix shows the before period estimated at 6.0 minutes per task by asking four staff to recall a typical case, and the after period observed at 4.85 minutes across 12 runs. The claimed saving is 1.15 minutes per task, or 207 minutes per day at your volume.

Now do three things. First, ask for the raw after times. If they run 3.9, 4.1, 4.2, 4.4, 4.6, 4.7, 4.8, 4.9, 5.0, 5.2, 5.4 and 11.0 minutes, the mean is 5.1 and one run accounts for most of the spread; with a tighter second sample the average could easily move by several percent. Second, replace the remembered baseline with one week of logged timestamps. Time studies are known to be vulnerable to outliers, and the usual remedy is repeat measurement across different combinations of people and equipment BSM Lean on the accuracy limits of direct timing. Third, check whether the after period counted human review time. If the agent's output went to a queue that took two minutes to clear, the observed task time understates the full cycle.

You can run this comparison on your own data in one afternoon. The point is not to catch the vendor out; it is to find out whether the number survives contact with your volume and your review step.

import statistics
after = [3.9, 4.1, 4.2, 4.4, 4.6, 4.7, 4.8, 4.9, 5.0, 5.2, 5.4, 11.0]  # minutes, invented
print("mean", round(statistics.mean(after), 2))
print("stdev", round(statistics.stdev(after), 2))
print("saving per day at 180 tasks", round((6.0 - statistics.mean(after)) * 180))

If you would rather have the baseline and the completion rules designed for you before a purchase decision, a free thirty-minute discovery call can scope the task list, the observation method and the acceptance criteria: book a discovery call.