How to Build a Golden Task Set for a Support Agent

Pull from four sources, in this order of trust:

1. Closed support tickets with a recorded resolution. Highest value, because the resolution is a human-verified outcome you can encode as the expected outcome.
2. Chat and voice transcripts that ended with a confirmed resolution or a confirmed escalation. Same value, harder to parse.
3. Tickets that were reopened within a short window. These are the failure-rich cases; a reopened ticket is a near-guaranteed hard task.
4. A small synthetic block for scenarios you know are coming but have not yet received, such as a policy change. Cap this at roughly one in five tasks so the set stays anchored to reality.

For each candidate you need to capture, at minimum: the opening message, the full conversation turns, the channel, the product and plan involved, the intent label, the gold outcome (what the correct final state is), and the grader notes. Provenance and reviewer identity belong in the record too, because audit-friendly datasets carry traceability metadata as recommended in the Maxim guide.

Two filters before sampling:

• Remove anything a customer would object to being replayed. Strip names, emails, card fragments and addresses, or replace them with stable placeholders. The task logic survives redaction; the identifiers do not matter.
• Remove anything the agent could not in principle resolve, such as tickets closed as duplicate or spam. They test nothing.

A worked number. Suppose your warehouse holds 40,000 closed conversations from two quarters and 2,400 reopened ones. The reopened ones are your difficulty fringe; the 40,000 are your base. You will sample deep from the reopened pool and shallow from the base, which is the inverse of what a uniform random draw would do.

Random sampling gives you the average ticket. Average tickets are where your agent already works. You want the distribution around the average. Stratify on four axes:

Worked example, aiming at 150 tasks:

• 45 percent single-turn resolution intents (about 68 tasks), spread across the six largest intents in proportion to volume.
• 35 percent multi-turn (about 52), with at least 20 of those at 5 or more turns.
• 15 percent difficulty fringe from the reopened pool (about 23).
• 5 percent adversarial (about 7), including at least two injection attempts and one contradiction.

Check the marginals before you freeze. If voice transcripts are 18 percent of real traffic but 40 percent of your set, either your agent has a voice-specific risk you are deliberately over-weighting (fine, document it) or you sampled wrong.

Volume matters for statistical confidence: with 150 binary pass/fail tasks, a 10-point shift in pass rate is roughly 15 tasks, which is more than the noise you would see with 30 tasks. Small sets produce swings that look like regressions but are just sampling. If two runs differ by fewer tasks than one task per point of rate change, do not act on it.

For a comparison-style view: DeepEval's dataset docs emphasize telling test coverage, varying complexity and edge cases, plus focused cases with clear scope. Read that as: breadth across strata, depth within the case.

Raw support logs are full of near-duplicates. The same "where is my order" ticket arrives 400 times with different order numbers. If all 400 enter the set, your pass rate becomes a measure of one intent, and a regression in a rare intent gets lost in the noise.

The procedure:

1. Normalize each candidate: lowercase, strip order numbers, dates, currency amounts and greetings.
2. Embed the normalized opening message plus intent label.
3. Cluster with a similarity threshold. In practice, cosine similarity above roughly 0.9 on sentence embeddings is a workable starting cut for "same underlying task"; tune it by eye against 30 pairs you have labeled yourself.
4. Keep one representative per cluster, preferring the one with the clearest human resolution and the most complete transcript.
5. Keep a second variant per cluster only when the resolution differs, for example same question, different plan tier, different refund eligibility.

This is the same logic behind log sampling: keep a representative fraction that preserves underlying behavior rather than every repeated entry as described in Groundcover's guide to log sampling.

Deduplicate again after every refresh. New tickets arrive in clusters, so the refresh cycle is also a deduplication cycle.

One trap: do not deduplicate on outcome. Two tickets with identical phrasing but different correct outcomes are the most valuable cases in the set, because they force the agent to read context. Keep both and flag them as a paired difficulty test.

A task without a grader is a prompt, not a test. Each task should carry several checks across three kinds of grader, and you should be explicit about which kind each check is:

• Code-based checks: deterministic, cheap, objective. Refund amount matches, ticket status is resolved, the correct tool was called with the correct order ID, the reply is valid JSON. Use these everywhere the truth is mechanical.
• LLM-as-a-judge checks: for tone, helpfulness, policy compliance in prose. Judges agree with human reviewers a large share of the time and remove reviewer inconsistency, but they inherit the prompt's flaws, so they must be calibrated against human labels per MLflow's judge guide and Evidently's guide to judge design.
• Human spot checks: a small slice of every run, to keep the judges honest. Google Cloud's framing puts human evaluation first as the ground truth that the automated layers are aligned to in its agent evaluation method.

A grading record worth copying:

task_id: refund-eligible-past-window-014
channel: chat
intent: refund_request
difficulty: multi_turn
source: reopened_ticket
opening: "I want a refund for order 88123, it arrived damaged."
gold_outcome:
  refund_status: approved
  amount_cents: 4299
  reason_code: damaged_on_arrival
checks:
  - type: code
    rule: final_refund_status == approved
  - type: code
    rule: refund_amount_cents == order_total_cents
  - type: judge
    rubric: "Agent acknowledges the damage without demanding photos it already received."
    fail_if: "agent asks for information present in the transcript"
  - type: judge
    rubric: "Agent does not promise a timeline longer than the policy states."

Two rules make rubrics survive contact with reviewers:

• Every check must be falsifiable by pointing at the transcript. "Agent was helpful" is not a check. "Agent did not ask for the order number a second time" is.
• One assertion per check. Bundled checks hide which failure you actually have. The components-of-evaluation model, with tasks carrying multiple graders each containing multiple checks, is spelled out in Anthropic's guide.

Also decide, in writing, what happens when the agent exceeds the rubric. Frontier agents sometimes solve a task better than the gold path, and a rigid grader will score that as a failure as Anthropic notes with a real flight-booking example. Your rule: the task author reviews any failed check flagged as "alternative valid path" within a set window, and either rewrites the check or records the case as a rubric defect.

Finally, freeze it. Version the corpus, tag each run with the corpus version, and store transcripts so a failed check can be re-examined later. Versioning and audit trails are what make the set a governance artifact and not just a folder per Maxim's governance guidance.

A frozen set drifts away from production. Products change, policies change, and customer phrasing changes. The fix is a schedule plus triggers, and it is the same principle behind datasets described as living artifacts that evolve with new failure modes and changing user behavior per the Maxim guide and the ISO/IEC 42001 alignment it cites.

Run the core on a fixed cadence (quarterly is a common default) and add tasks when any of these fire:

The decision rule for replacing rather than adding: if a task's gold outcome is no longer what a human would do today, retire it from the active set and move it to an archive. Never delete it. Archived tasks are your regression suite for the old policy, and if the policy ever returns you have your bar back.

Keep the archive as a second suite, clearly labeled, run less often. Then the active set stays small enough to review and the archive stays large enough to catch backsliding.

Schedule-wise: the refresh job is a recurring piece of work with an owner, a review step by someone who reads support tickets, and a change log entry. If it has no owner, it will not happen after the second cycle. Provenance and reviewer records are again the mechanism that makes this auditable per Maxim.

One more maintenance note: keep the set out of any prompt or fine-tuning pipeline. A model that has seen the tasks is no longer being tested by them, and contamination is the fastest way to a launch bar that always passes as flagged in the decontamination principle.

If your support agent needs outputs from several model families for judging and generation across the suite, running them behind one key removes credential churn from your harness: Quanta Inference exposes text, image and audio models through one OpenAI-compatible endpoint.