How to Evaluate an AI Agent Before Launch

The test set does not need to be large to be useful. Anthropic's guide says teams often delay evals because they think they need hundreds of tasks, when "20-50 simple tasks drawn from real failures is a great start." Early in development each change has a large effect, so a small set is enough to see it (Anthropic).

Where to find the first tasks:

1. The checks you already run by hand before each demo or release.
2. Support tickets and bug reports if anything is already in production.
3. Requests from people who will use the agent, written in their own words, not cleaned up by engineers.
4. Adversarial inputs: prompt injection attempts, off-topic requests, angry users and missing information.

Each task needs inputs, an expected outcome and, ideally, a reference solution that you have confirmed passes your graders. Anthropic's test for task quality is that two domain experts would independently reach the same pass or fail verdict. If they would not, the task is ambiguous, and ambiguity in tasks becomes noise in your scores.

Balance matters as much as size. Anthropic describes building search evals for its own assistant: if you only test cases where the model should search, you end up with an agent that searches for everything. Include cases where the right behavior is to not call a tool, not take an action or hand off to a person (Anthropic).

Langfuse recommends scoping each dataset to one question, such as end-to-end support or retrieval alone, and sizing it to the job. Its working guide suggests around 10 items to explore a single issue and 100 to 1,000 items for checks on larger changes that should cover the production distribution. It also warns that a dataset of easy cases "scores 95 percent forever and tells you nothing" (Langfuse, Golden dataset evaluation).

The most dangerous agent failures produce a good-looking final message. Confident AI describes several patterns that output-only scoring misses (Confident AI, AI agent evaluation guide):

• The ghost action. A booking agent replies "Done, your flight is booked" but never called the booking tool. The message is fluent and relevant, and nothing happened.
• The interrogation loop. A user gives an order number, and the agent keeps asking for it, because it never maps the number into the tool's parameter.
• The confident fabricator. A research agent fills gaps in stale search results with its own guesses and presents them as current facts.
• The budget burner. The task completes, but only after many redundant model calls that multiply cost.

To catch these, separate two things Anthropic distinguishes clearly. The transcript is the full record of a run: messages, tool calls, reasoning and intermediate results. The outcome is the final state of the world. "A flight-booking agent might say 'Your flight has been booked' at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database" (Anthropic).

So for each task, check the outcome directly where you can: query the database, read the file, inspect the ticket status. Then check the trace for things the outcome cannot show: forbidden tool calls, too many turns, excessive tokens or a step that leaked data.

Google Cloud calls the reverse case a silent failure: an agent reports the right inventory number but read it from last year's report. The answer is correct and the process is wrong, and only the trajectory reveals it (Google Cloud).

One caution. Do not require one exact sequence of tool calls. Anthropic found that approach "too rigid," because agents regularly find valid paths the designers did not anticipate. Grade what the agent produced and the rules it must not break, and give partial credit for multi-step tasks where it got some of the way.

There are three kinds of graders, and a good eval mixes them. The table below summarizes Anthropic's comparison (Anthropic):

The rule of thumb is to use code wherever a check can be written as code, a model where it cannot and humans to set the standard and check the model graders. Google Cloud recommends the same layering and adds a fourth category, adversarial testing, to see whether the agent fails safely under hostile input (Google Cloud).

A single support task might combine all of them:

task: refund-damaged-item
input: "My blender arrived cracked. Order 4821. I want my money back."
graders:
  - type: state_check            # code
    expect: { refunds: { order: 4821, status: processed } }
  - type: tool_calls             # code
    must_include: [verify_identity, process_refund]
    must_not_include: [delete_account]
  - type: transcript             # code
    max_turns: 8
  - type: rubric                 # model
    checks:
      - "Explains the refund timeline in plain words"
      - "Does not promise anything the policy does not allow"

A model that grades another model is only as good as its prompt and its calibration. Evidently AI describes building a judge as "a small ML project": label a set of examples yourself, write the evaluation prompt, run it on your labels and measure agreement with metrics such as precision and recall before trusting it (Evidently AI, LLM-as-a-judge guide).

Practical rules from the sources:

• Keep each judge to one question. Evidently recommends splitting dimensions such as tone and accuracy into separate evaluations and using clear binary choices where possible. Anthropic likewise recommends grading each dimension with an isolated judge rather than one judge for everything.
• Give the judge a way out. Anthropic suggests instructing it to return "Unknown" when it lacks the information to decide, which reduces invented verdicts.
• Keep a held-out set. Evidently recommends reserving part of your labeled data to test the final judge prompt, so you do not tune the judge to the same examples you measure it on.
• Bring in domain experts. The people who know what a good answer looks like should write the criteria and review disagreements.
• Recalibrate. Expectations shift as you read more outputs, a pattern Evidently calls criteria drift, so re-check the judge against fresh human labels periodically.

Google Cloud puts the order of operations plainly: humans identify the failure modes first, then the judge is built to detect them and aligned against the human grading before deployment (Google Cloud).

Agents are not deterministic. The same task can pass on one run and fail on the next, so a single pass tells you little. Anthropic calls each attempt a trial and recommends running several per task. It also describes two metrics for summarizing them (Anthropic):

• pass@k is the chance that at least one of k attempts succeeds. It suits tools where one good answer is enough, such as generating candidate solutions.
• pass^k is the chance that all k attempts succeed. It suits customer-facing agents, where every user expects it to work.

The gap between them can be large. If a task succeeds 75 percent of the time, the chance of three successes in a row is 0.75 cubed, about 42 percent. An agent that "usually works" can still fail a large share of customers.

Keep the test environment clean between trials. Anthropic warns that shared state, such as leftover files or cached data, can cause correlated failures or inflate scores. In one internal case, a model gained an unfair advantage by reading the git history left by previous trials.

Decide the bar before you see the results, so the numbers cannot talk you into launching. A reasonable bar names a minimum pass rate per suite, zero failures on safety and permission checks, and limits on turns and cost per task. Anthropic separates two kinds of suite: capability evals, which start with a low pass rate and give the team something to climb, and regression evals, which "should have a nearly 100% pass rate" and catch anything that breaks (Anthropic).

Before launch, read transcripts, not only scores. Anthropic's rule is not to take eval scores at face value until someone has read some transcripts, because a failure might be the agent's fault or a broken grader. It also notes that a 0 percent pass rate across many trials is usually a sign of a broken task rather than an incapable agent.

After launch, the evaluation set becomes a living record. Langfuse recommends routing flagged production traces, such as thumbs-down feedback, into the dataset as candidates, having an expert write the corrected expected output, deduplicating near-identical items and versioning the dataset so scores from different dates stay comparable (Langfuse).

A short pre-launch checklist:

• Success statements are written and each maps to at least one check.
• The task set includes real requests, past failures and cases where the agent should decline or hand off.
• Outcome checks verify state, not just the agent's claims.
• Model judges are calibrated against human labels and a held-out set.
• Every task runs several trials, and customer-facing flows are judged on consistency.
• The launch bar was written down before the final run.
• Someone has read a sample of transcripts from the final run.

The same suite earns its keep after launch. Rerun it before every prompt change, tool change or model swap, including when you move an agent to a different model through a gateway such as Quanta Inference.