How to Test Whether an Agent Performs a Task as Fast as a Human

Timing only successful runs, or timing everything and calling it speed, will both mislead you. Atlan's writeup on agent task success draws the distinction plainly: task completion is an execution signal, task success is an outcome judgment, and an agent can return a response, finish its workflow and log no errors while still failing the user's request Atlan on task success rate. The same piece reports that Salesforce found nearly one in seven interactions that looked successful in system logs had failed the user, a pattern it calls phantom successes.

So record four fields per run, not one:

1. started_at and ended_at (wall clock, plus a separate model or compute time if you can get it)
2. outcome: pass, fail, or timeout
3. attempt: 1 for first try, 2 or more for retries
4. setup_seconds: provisioning, credentials, prompt or context assembly

Then compute speed on two populations: first-attempt passes, and all runs including retries. Report both. A tool that is fast when it works and fails a third of the time is not faster in production.

The most common mistake is timing three tasks with one enthusiastic colleague and calling it a baseline.

MeasuringU's guidance on sample size for task times treats time as a continuous measure with its own variability, which means the number you need depends on how spread out your times are, not on a fixed rule of thumb MeasuringU on sample size for task times. Roughly, you need more runs when times vary widely and fewer when they cluster tightly. Thirty per side is a defensible floor for a first look; it gives you a usable median and a crude 90th percentile.

If you also want a pass rate (fraction of runs that succeed), that is a proportion, and proportions near 0 or 1 need care. Disney reviewed binomial confidence intervals for activity sampling and found the widely used Wald interval performs poorly, especially when the observed probability is near zero or one, and recommended the Clopper-Pearson interval instead Disney on binomial confidence intervals. Practical translation: if your agent passes 19 of 20 runs, do not report "95% success". Report the interval, which is much wider than intuition suggests.

On the human side, use several people, not one. Time study practice warns that a single stopwatch reading risks measuring an outlier, and the fix is repeated measurement across different combinations of people and equipment BSM Lean on time studies. Aim for at least five people on the human side, each doing the same sampled task instances. That lets you see whether your "human baseline" is really one fast person's personal best.

Worth knowing before you design the human arm: work sampling and continuous time-and-motion measurement do not agree closely. A study of medical residents found that work sampling results differed by 20 percent or more of the estimated value for eight of ten activities compared with continuous time-and-motion observation PubMed comparison of work sampling and time-and-motion. If you sample human behavior intermittently instead of timing whole tasks, expect that much error.

Timing drift comes from measuring the agent with API logs and the human with a stopwatch held by a colleague who started it late. Put both through one wrapper.

The wrapper does four things: writes a start row, runs the task, writes an end row with outcome, and never lets a run exceed the timeout.

import time, uuid, json

def run_timed(task_id, actor, actor_type, fn, timeout_s=1800):
    run_id = str(uuid.uuid4())
    t0 = time.time()
    record = {"run_id": run_id, "task_id": task_id, "actor": actor,
              "actor_type": actor_type, "started_at": t0}
    try:
        result = fn()                      # agent call or human-submitted artifact
        outcome = "pass" if result["ok"] else "fail"
    except TimeoutError:
        outcome = "timeout"
    except Exception as e:
        outcome = f"error:{type(e).__name__}"
    record.update({"ended_at": time.time(), "outcome": outcome,
                   "elapsed_s": time.time() - t0})
    with open("runs.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")
    return record

For humans, fn is a form submission: the person clicks "start" when they open the task and "submit" when they attach the finished artifact. That makes the human timestamp come from the same clock and the same table as the agent's. It also removes the observer effect that makes stopwatch studies feel adversarial, which practitioners flag as a real risk to data quality BSM Lean on work measurement.

Two instrumentation details that change conclusions:

• Include waiting time. If the agent queues behind a rate limit for 90 seconds, that is user-perceived latency. Log queue time separately but count it in the headline.
• Exclude tool setup from task time, and report it. Provisioning an agent can take days; a person already has a laptop. Mixing them flatters whichever side you set up first.

If your agents run across several model providers, routing them through one endpoint keeps the wrapper simple, which is the case Quanta Inference is built for: https://quantaintelligence.ai/inference.

Task times are skewed: a few runs take far longer than the rest, and means get dragged around by them. Report the median (typical), the 90th percentile (what your worst common case feels like), and the ratio between the two (spread). That trio describes the experience far better than one average.

Here is a worked example with 30 runs per side, in seconds:

Read that table honestly. The agent is more than twice as fast in the typical case, but slower than a person at the 90th percentile, and its spread is more than three times wider. If you only reported the medians you would conclude "agent wins by 2.3x". If you only reported means, one 30-minute timeout would swamp the result.

That last row matters more than it looks: setup added 50 seconds to the agent's median here, which is small, but on a five-minute task it would be a 17% tax on every run. Keep it visible.

Choose the summary before you collect data, and choose the test before you look at results, which is standard practice in study design generally Nayak and Hazra on choosing a statistical test. For skewed times, compare medians with a nonparametric test such as Mann-Whitney U, and report the difference with a confidence interval rather than a bare p-value. The UK's Office for National Statistics explains why: standard error, confidence interval and coefficient of variation exist because a sample estimate will differ from the true value, and the interval is how you communicate that uncertainty ONS on uncertainty and how we measure it. Appinio's overview covers the same mechanics from a market research angle, including how confidence level and sample size trade off against precision Appinio on confidence intervals.

One more number worth publishing: time to first human touch. Agents that finish unattended in 20 minutes but need 15 minutes of human review are not 20-minute tasks.

Before you run anything, write the decision rule. A useful one:

Adopt the agent for this task if the agent median is at most 50% of the human median, the agent 90th percentile is at most the human 90th percentile, and first-attempt pass rate is within 5 points of the human rate.

Three conditions, because speed alone is a bad adoption criterion. An agent that is fast on average but produces a long tail of slow failures creates unpredictable work for whoever supervises it.

Consider cost of poor quality when setting the pass-rate bar. Quality cost frameworks split spending into prevention, appraisal, internal failure and external failure, and treat rework, scrap, failure analysis, warranty claims and returns as the expensive categories once work has to be redone SixSigma.us on COPQ. Translated to agents: an agent that is twice as fast but needs rework on one in five outputs may still win internally, and lose badly if those outputs go straight to a customer. LeanMap's failure cost calculator is a simple way to put a number on rework per bad run LeanMap cost of poor quality calculator.

Time studies also give you a vocabulary for the human side of this rule. A standard time in work measurement is the time a qualified worker takes under defined conditions at a designated performance rate, with allowances for delays and breaks folded in deliberately rather than hidden SixSigma.us on time study analysis. If your human baseline omits the interruptions a real day contains, the agent will look slower than it is against a fantasy human.

Some tasks resist clean timing: long, ambiguous, or dependent on context the agent cannot see. METR's answer was to change the unit of measurement from "how fast" to "how long a task the agent can finish at a given reliability", timing human experts on the same tasks and fitting the probability of success against human task duration METR's paper on measuring long tasks. Their headline finding is that current models succeed almost always on tasks that take humans under four minutes and rarely on tasks that take humans more than about four hours METR's research summary.

That framing is more useful than a stopwatch comparison for planning. It tells you where on your task list to expect the agent to hold up. But METR is explicit about the limits: their human estimates likely overestimate expert time because the humans had less context than they would normally have, and measurements above their longest task length are unreliable METR's time horizons page. Your internal study will have the same bias unless you let your humans work with their normal context.

If your task is long and multi-step, measure in two layers: median time per step, and fraction of full workflows completed inside the timeout. The first diagnoses, the second decides.

A final note on scope. A task that looks atomic often hides selection behavior: research on hospital emergency departments found that under heavier workload people complete easier tasks first, which lifts short-term throughput while hurting longer-run speed Harvard Business School on task selection and workload. Agents with a queue of tasks can exhibit the same pattern, so fix the order of task instances in advance for both sides.

For teams that need a defensible baseline before committing budget, a structured review of your task list and measurement plan is often faster than a month of ad hoc timing, which is what a free discovery call is for: https://quantaintelligence.ai/calendar.