How to Compare Two Agent Vendors on the Same Task Without Bias

The sample is your experiment. Treat it like a test set: frozen, documented and never tuned to either vendor.

Pick 20 to 50 tasks from real work. Sample from the same case mix your production workflow sees, including the exception categories you already know about. If 30 percent of real tickets are refund disputes and 10 percent are account merges, the sample should reflect that. A sample of only clean, easy cases flatters both vendors and tells you nothing.

Write a gold answer for every task. For each task, record the correct outcome, the required steps, and the disqualifying errors. If a human expert cannot state the gold answer in a few lines, the task is not ready for scoring.

Size the sample against the effect you need to detect. Maxim AI notes that agent quality A/B tests typically need far more trajectories than conversion tests to separate a real lift from model stochasticity, on the order of 10,000 or more per arm when the metric is subjective like helpfulness or factuality (Maxim AI). You are not running that test. You are making a purchase decision, so accept the trade-off explicitly: 20 to 50 tasks will find a large difference and will miss a small one. That is the right sensitivity for choosing a vendor, and it is the wrong tool for claiming a 3 percent quality lift.

Freeze the sample before the first run. Store it as a CSV with a task ID, the input, the gold answer and the acceptable-variation notes. Everything downstream joins on that ID.

That last row is the insight most evaluation plans miss. Keep 5 to 10 tasks unseen and run them once, at the end, on the winner. If its score collapses on the held-out set, you were measuring prompt tuning, not capability.

Every environmental difference becomes an alternate explanation for the result. Close them all.

• One prompt template. Same system instructions, same user wrapper, same output format request. If a vendor needs a different wrapper to function, document it and use the identical wrapper for both.
• Identical tools. Same APIs, same data sources, same retrieval index, same permissions. Maxim AI lists the control arm drifting as a core problem in agent testing: even if you do not change the variant, the underlying model, retrieval index or tool definitions will, and a test against a moving baseline tells you very little (Maxim AI).
• Same time limit per task. A vendor given ten minutes will beat one given one minute for reasons unrelated to the product.
• Same model tier. If one vendor lets you choose the backing model, pin the tier for both. OpenAI's agent guide makes the related point that you should establish a performance baseline with the most capable model and then swap in smaller models only where accuracy holds (OpenAI). Here, baseline means "same tier for both arms", not "best tier I can afford for one".
• Run interleaved, not back to back. Run vendor A on tasks 1 to 10, then B on 1 to 10, then A on 11 to 20, and so on. If a provider has a bad hour, both arms feel it.
• Log everything. Task ID, vendor, timestamp, latency in seconds, token or unit consumption, tool calls made, and any errors. You cannot compute cost per successful task without the consumption log.

A simple run manifest keeps this honest:

sample_version: tasks-v3-frozen.csv
prompt_template: agent-eval-v2.txt
tools: [crm_read, refund_api, ticket_update]
time_limit_seconds: 300
model_tier: same_for_both
run_order: interleaved
held_out: tasks-heldout-v1.csv

When a colleague asks "was this fair", the manifest is the answer.

This is the step that separates a comparison from a demo contest, and it is the one most teams skip.

Vendor demos are persuasive because of theater: a polished presenter, a scripted scenario, a warm room. A vendor evaluation rubric exists precisely to force leaders to name what matters first, because a flashy demo can bend the room fast, and a scorecard that only tracks points without standards behind the points produces a clean-looking score on top of a messy decision (Cloud Tech Gurus).

Blinding removes the person from that equation. Procedure:

1. Export every output as plain text or a rendered artifact.
2. Strip vendor names, model names, headers, footers, logos, distinctive formatting and any self-referential phrasing the agent used.
3. Randomize the order across both vendors using a script, and record the mapping in a separate file only the study owner reads.
4. Assign each blinded output a neutral ID such as OUT-0147.
5. Hand reviewers a sheet with output ID and task ID only.

If the outputs have different shapes, for example one agent returns a structured plan and the other returns a chat transcript, normalize on the artifact that matters to you, which is usually the final action taken or the deliverable produced, and keep the transcript as a secondary artifact. Do not let a formatting difference become the score.

The review tooling matters less than the separation of duties. Two reviewers, working alone, before any group discussion. The Cloud Tech Gurus rubric guidance says it plainly: if two reviewers score the same demo two points apart, the vendor is not the problem, the rubric is (Cloud Tech Gurus).

One number per output is not enough. Score a handful of dimensions, then compute the average. A workable set for agent tasks:

Score each dimension 1 to 5, then report two numbers: the mean across dimensions for each output, and a separate binary success flag. The flag answers "would we ship this to a customer" and should be defined before scoring, for example "correctness of 4 or higher and no policy violation". Vendor selection frameworks in procurement make the same split between scored levels and must-have gates, with weights set before demos and reviewers calibrated on one sample vendor first (Cloud Tech Gurus). The IT vendor criteria checklist literature reaches a similar conclusion: define must-haves, deal-breakers and evidence requirements up front to prevent scope creep and demo-driven bias (Technology Match).

A worked example, with invented numbers:

• 30 tasks, two vendors, two reviewers, 120 scored outputs.
• Reviewer disagreement: on correctness, the two reviewers differ by 1 point or more on 12 of 60 outputs. Half of those are 1-point differences, so use the absolute disagreement as your noise estimate rather than its square.
• Mean correctness: vendor A 3.84, vendor B 3.51.
• Mean success rate on the binary flag: A 70 percent, B 57 percent.
• Consumption log: A used 41 units on 30 tasks, B used 33.

That data suggests A is better on quality and worse on cost. The next section turns it into a decision.

Price per token or per call is the number vendors quote and the number buyers compare. It is the wrong number, because a cheap agent that fails half the time is expensive.

Cost per successful task equals total spend divided by the count of outputs that passed the binary success flag.

Using the example: if one unit costs a dollar in this illustrative budget, A spends 41 dollars for 21 successes, about 1.95 dollars per success. B spends 33 dollars for 17 successes, about 1.94 dollars per success. The vendors are statistically tied on unit economics, and A's quality advantage becomes the deciding factor, provided it survives the noise test.

Change one input and the ranking flips. If B's failures are concentrated in the tasks that trigger heavy retries, its consumption on successful tasks may be much higher than the total suggests, which is why you keep per-task logs and not invoices. Run the calculation per task ID and check whether the distribution is skewed by a handful of outliers.

Do the same for human review time, since that is often the real cost. WIQ's ROI guidance for agent automation says to include human review, exception handling and maintenance in the model rather than counting only compute (WIQ). Practical version: track minutes a reviewer spends correcting each failed output, multiply by a loaded hourly rate, and add it to the vendor's spend before dividing by successes. An agent that saves a dollar per task but costs four minutes of rework per failure is usually the more expensive option.

The most common error in vendor comparisons is treating any score difference as a finding. It is not. You measured reviewer disagreement; that is your noise.

Set the rule before you see the scores:

Prefer the higher scorer only if the gap exceeds your measured disagreement. If reviewers routinely differ by 1 point, a 0.2 point lead is not a result. If your binary success flags agreed on 90 percent of outputs, a 13 point gap on a 30-task sample is worth acting on.

Use a simple paired comparison. For each task, compute the difference in score between the two vendors. Count how many tasks favor A, how many favor B, and how many are tied. Report those three counts. If A wins 18 tasks, B wins 6 and 6 are tied, that is a convincing signal at this sample size. If it is 12, 10 and 8, you have a tie and should decide on cost, support or risk instead of quality.

Keep a decision log. Record the pre-registered threshold, the results, and the reasoning. The procurement literature is blunt about why: a structured evaluation creates an audit trail your security, finance and legal teams can stand behind, and ties requirements to scores and scores to rationale (Technology Match). A vendor selection process that produces a defensible record also gives you leverage on contract terms, service levels and exit clauses (Ivalua).

Do not skip the non-quality factors. Quality scores answer one question. Rollout risk, security posture, integration effort and support obligations answer the others, and a rubric that weights all of them can change the winner even when quality is clear. Jimmy's team in the contact-center case above watched an incumbent lose on features while a challenger lost on rollout depth, and the weighting exposed the real risk (Cloud Tech Gurus). Add your own weights for integration effort and support, set them before scoring, and present both the quality-only view and the weighted view. If they disagree, that disagreement is the meeting agenda.

One last check on assumptions. The business planning literature is consistent that assumptions are educated guesses that will sometimes be wrong, and that documenting them lets you revisit and correct them instead of rebuilding from memory (John Olivant). Write down what you assumed about task volume, review time and failure cost. When the winner goes live, you can test those assumptions against real numbers instead of arguing about a pilot deck.