How to Compare an AI Agent's Output to the Work It Replaced

Start with a list, not a score. Enumerate every output item the workflow produces over a fixed window: documents, tickets, code changes, reports, replies, analyses. For each item record five fields.

The acceptance criteria column is the one people skip, and it is the one that makes or breaks the comparison. A pilot with a quantified success definition agreed up front achieves a much higher success rate than one without; research summarized by the Institute of AI PM puts that at 54 percent versus 12 percent (Institute of AI PM). Writing the criteria down per item, before the AI touches that task type, is how you inherit that advantage.

The inventory also gives you the denominator. A pilot that wins on five hand-picked tickets and loses on fifty unremarkable ones has not won. The AI Engineer's evaluation guidance is direct about keeping denominators visible beside performance numbers (AI Engineer). If your inventory covers 200 tickets and your comparison covers 30, say so on the same line as the accuracy figure.

A practical shortcut: run the inventory for two comparable weeks, one before the AI was introduced and one after, and list items in the same categories both times. This is the same logic as a stage gate, where the move from one phase to the next requires evidence against agreed exit criteria rather than a good impression (Rubicon).

For each AI-produced item, find the human-produced counterpart that matches on all four of these:

1. Task type. A refund request is not a cancellation. A bug fix in a payments module is not a CSS tweak. Group by the decision or action the output drives, not by the artifact format.
2. Difficulty. Stratify deliberately. Use a simple, typical, edge-case split and make sure each AI item is paired within its stratum. Easy inputs tell you the happy path works; they do not tell you whether the approach is dependable (SOLTECH).
3. Inputs. Same source material, same time window, same data state. If the AI saw a cleaned export and the human saw live messy data, you are measuring the data, not the agent.
4. Acceptance criteria and review depth. Same definition of done and the same number of reviewer passes. If AI output gets a light skim and human output gets a full review, the accuracy figures mean nothing.

Matching on all four is stricter than most pilots manage, so use a pairing table and score the match quality explicitly.

The point of the pair quality column is honesty, not perfection. A flagged pair still contributes evidence, but it is reported separately and never pooled silently into a headline number. When you present results, report full pairs and flagged pairs as two lines. This mirrors good practice in human performance measurement, where the emphasis is on choosing measures with demonstrated reliability and sensitivity rather than aggregating whatever numbers are available (PubMed).

Write the checklist once, before scoring, and apply it identically to both sides. Use the same reviewers for human and AI output where possible, and hide provenance from them if you can. A checklist for a support reply might read: correct policy applied, correct facts cited, tone appropriate, no commitment made beyond authority, resolution achievable without a second contact.

For code changes the checklist shifts: does it fix the stated defect, do tests cover the change, does it touch unrelated code, would a reviewer approve it as is. For reports: are the numbers traceable to source, are there unsupported claims, is the recommendation actionable.

Two rules make the checklist trustworthy.

• Score the outcome, not the artifact. The question is whether the work was accepted, edited, reworked or escalated, not whether the prose reads well. Structured human review scales when it uses clear rubrics, consistent scoring criteria and periodic calibration so reviewers stay aligned (Label Studio).
• Score in meaningful slices. Break results by task type, difficulty stratum, language, region or whatever segment maps to risk (Label Studio). A single blended score hides the fact that the agent is strong on typical refunds and weak on policy exceptions.

On the four metrics, define them once and stick to them. Accuracy is the share of items passing every checklist line without edit. Time is wall-clock from start to accepted, including review. Rework is the count of items returned for at least one substantive edit. Escalation is the count of items that needed a supervisor, a second team or a rollback. All four attach to each item in both columns of the pair.

Plenty of AI work has no human predecessor: a new summarization feed, an agent answering a long tail of questions nobody handled before, a monitoring task that simply was not done. There are three defensible baselines.

Comparable historical task. Find the closest existing work. If the agent now drafts renewal notices, look at the most similar notice type a person already drafted, and match on the four dimensions in Step 2. State plainly that this is a proxy. Historical proxies carry the risk of criteria drift, which is why the acceptance criteria column in your inventory matters.

Documented process standard. Where a written standard exists (a style guide, a policy, a code review rubric), score both the AI output and a human output against it. This is the most stable baseline because the standard does not move between scoring sessions. The education sector version of this logic is well documented: districts set baseline metrics before launch, then calculate deltas at review, precisely so that novelty bias does not pass for improvement (SchoolAmplified).

Have a person perform the task under the same conditions. The cleanest baseline, and the most expensive. Give a real practitioner the same inputs, the same acceptance criteria and the same time budget, then score both. This is what makes the comparison like-for-like by construction. The human performance literature is useful here: choose measures that are reliable and sensitive to the condition you are testing, and accept that many commonly used measures have not been validated for reliability in this kind of setting (PubMed).

One warning on the third route. Human performance is not a single number, and traditional activity metrics such as hours worked or output produced capture only the organization's view, not the quality of the work (Deloitte). Pair each human baseline item with the same checklist you use for the AI side, or you will compare a rich AI scorecard against a thin human one.

When you genuinely cannot build any of the three, the honest move is to label the comparison "no baseline" and report only within-AI variation or against thresholds you set in advance. A threshold set before seeing results is far weaker evidence than a paired human comparison, and the write-up should say so.

A support organization runs 400 first-response tickets over two weeks. Before the pilot it ran 360 human-handled tickets over a comparable two weeks, from the same queues, with the same policy version. The inventory covers both windows with the five fields from Step 1.

Pairing produces 240 full pairs (AI item and human item matched on all four dimensions) and 40 flagged pairs. The remaining AI items had no human counterpart at that difficulty level and are reported separately.

The naive read is that the AI is faster but slightly less accurate and escalates more. The useful read comes from slicing. Suppose the slice by difficulty shows that on typical tickets the AI passed the checklist at 72 percent against 69 percent for humans, while on edge cases it passed at 51 percent against 74 percent, and that escalations concentrate in the edge stratum. That changes the decision: route typical work to the agent, keep edge cases with people, and send every escalation back into the pairing table as a new test case. This is the practical application of evaluating by meaningful slices rather than a single blended score (Label Studio).

Then check the flagged pairs separately. If flagged pairs show 23 percent escalation while full pairs show 12 percent, some of the apparent escalation could be match quality rather than agent weakness. Reporting both lines protects you from a conclusion you cannot defend.

Two more numbers belong in the write-up. First, the count of AI items with no human counterpart, because they represent the work that previously did not happen and should not be credited as efficiency without a value argument. Second, an attempt policy: state whether a pair counts when the AI was retried, and how many attempts were allowed (AI Engineer). An agent that takes three tries to pass the checklist is not equivalent to a human who passed on the first.

The cleaned-input mismatch. The AI saw a tidy export and the human saw live data. Catch it in the inputs column of the pairing table. If two-thirds of your pairs are flagged on inputs, you do not have a comparison yet.

The review-depth mismatch. AI output gets a skim; human output gets a full review. This usually shows up as implausibly strong AI accuracy and a suspiciously low rework rate on the human side. Fix it by rotating reviewers across both sides without provenance.

The criteria drift. Policy changed between the two windows, so the human and AI items were accepted under different rules. The Institute of AI PM's diagnostic question is worth borrowing: show me a live sample and walk me through who owns it and how it arrives (Institute of AI PM). If the answer involves a spreadsheet or a manual step, expect drift.

The denominator disappearance. A headline of 90 percent accuracy drawn from the easy stratum. Always print the denominator and the stratum beside the headline (AI Engineer).

The silent human patch. In pilots, people quietly fill context gaps and absorb risk. Production removes the patch and the agent falters (UnifyApps). Pair data collected during the pilot therefore flatters the agent. Capture rework and escalation from the first weeks of live operation, not only from the pilot.

If your team is running several agents across several task types and the inventory work is the bottleneck, that is exactly the kind of structuring work where a line-of-business expert can do the pairing and the scoring and hand back the finished comparison. Quanta Experts is built for that: https://quantaintelligence.ai/agents. And if you are still choosing which model endpoint to run each task type against, Quanta Inference puts text, image, video, audio and embedding models behind one API key: https://quantaintelligence.ai/inference.