Synthetic Tasks: Generating Cases You Have No Logs For

Before generating anything, produce a short list of slices with counts. Pull traces over a fixed window, tag each session with your intent classifier or a human-assigned label, and count. Then sort by two numbers: volume and coverage in the golden set.

The decision rule: if a slice has fewer than roughly 20 items in the golden set, you cannot compute a per-slice score with any confidence, so it needs synthetic items. If it already has 20 to 40, prefer real items, because real logs carry the mess you cannot invent.

Two practical notes. First, for agentic systems, count steps and tool calls, not just final answers, because a single request can trigger dozens of internal decisions and the interesting failure lives in the middle of the trajectory LangChain. Second, include an "angry customer" slice even if your classifier does not have it: sentiment is often a separate signal from intent, and frustration is exactly where policy compliance and tone both get tested at once MLflow.

A spec is the difference between a synthetic task and a synthetic sentence. It is a small structured record, not free text, because downstream rule filters need fields to check. Write one per gap, and keep it under 15 lines.

{
  "spec_id": "refund-angry-003",
  "intent": "cancel_subscription",
  "sub_intent": "refund_after_failed_payment",
  "context": {
    "customer_tenure_months": 14,
    "payment_failures_in_30d": 2,
    "prior_agent_turns": 3,
    "sentiment": "angry",
    "channel": "chat"
  },
  "expected_behavior": "Acknowledge frustration, confirm the two failed charges, offer refund of the last charge only, state the refund timeline, offer a human handoff.",
  "hard_constraints": [
    "must not promise a refund beyond 30 days",
    "must not blame the payment provider by name",
    "must include a human handoff offer"
  ],
  "must_not": ["apologize more than twice", "ask the user to repeat account details"],
  "difficulty": "medium",
  "source_pattern_ref": "trace_id:8f3a21c"
}

The expected_behavior field is your definition of success. Google's agent evaluation guidance makes the point that vague goals like "the agent should be helpful" lead nowhere, while a specific measurable statement of success converts directly into metrics Google Cloud. A spec per gap forces you to write that statement before you see any generated text, which stops you from rationalizing whatever the model produces.

Keep source_pattern_ref pointing at at least one real trace that resembles the gap. You are not copying it; you are anchoring the spec to something that actually happened, which is what makes the later real-log pattern check meaningful.

Generate 5 to 10 candidates per spec, not one. Diversity within a spec is what stops you from trading one kind of overfitting for another, and it gives the filters something to reject.

Practical settings that matter more than the model choice:

• Temperature around 0.8 to 1.0 for candidate variety. You want the model reaching for different phrasings of anger, not the same sentence five ways.
• A generation prompt that receives the spec as data and is told to vary phrasing, length, typos, lowercase, and slang while preserving every hard constraint.
• Batch by spec rather than by intent, so each call has one behavior to satisfy.
• Log the generation prompt version with every candidate. When a filter starts rejecting 60 percent of output, you need to know which prompt produced it.

If you are generating and evaluating across several model families, running them behind one key removes a whole class of plumbing work; Quanta Inference puts text, image, video, audio and embedding models behind a single OpenAI-compatible endpoint (https://quantaintelligence.ai/inference).

One caution on multi-turn candidates. If the spec involves a conversation, generate turns rather than a single message, and store them as a scenario plus expected outcome, the way conversational goldens are defined, so that evaluation can simulate the dialogue rather than match one string DeepEval.

Verification is the part people skip, and skipping it is how synthetic tasks corrupt scores. Four checks, in this order, each cheaper than the next.

Rule filters first. Deterministic checks belong in code, not in a model: valid schema, no leftover template tokens, the hard constraints present as substrings or regexes, sentence length within the range you observe in real traffic. Google's framework calls code-based evaluations the cheapest and most deterministic layer and recommends them for exactly this kind of checkable failure Google Cloud. Expect this step alone to reject a meaningful share of raw candidates, mostly for missing a constraint clause.

Then real-log pattern checks. This is the step that separates synthetic tasks you can trust from synthetic tasks that read like a language model wrote them. Compare each surviving candidate against the distribution of the real slice it is meant to represent: average message length, punctuation habits, presence of filler words, how often the customer mentions an order number, how often anger shows up as profanity versus clipped sentences versus threats to leave. If your fake angry customers all write complete grammatical sentences with an exclamation mark, and your real ones write three words in lowercase, the model will learn nothing useful and the slice score will be meaningless. A simple check: embed candidates and real items, and flag any candidate whose nearest real neighbor is farther away than the 95th percentile real-to-real distance.

Then human review on a sample. Sample 20 to 30 verified candidates per slice and have a domain expert label them as realistic or not, and as correctly specified or not. This is where you find the subtle killer: a candidate that satisfies your stated policy but makes a commitment your business would never honor. LLM judges agree with human reviewers about 85 percent of the time, which is good for monitoring at scale, but that also means roughly one in seven judgments disagrees with a careful human, so a judge alone is not a sufficient gate for admitting new golden items Confident AI. Use the judge to triage volume and humans to settle the boundary.

Then the score comparison. This is the decisive test and it is the one most teams never run. Take your held-out real set (real items, never used in tuning, never touched by generation). Score your system twice: once on real-only items, once on a combined set that includes the new synthetic items. If the combined score moves by more than a couple of points in either direction, your synthetic items are not neutral replacements for the real slice. They are easier, or harder, or they encode the generation model's stylistic biases. Investigate before admitting them.

A worked example with numbers makes the risk concrete. Say your real-only golden set scores 0.82 on your correctness judge. After adding 120 synthetic items for angry refunds, the combined set scores 0.61. That 21-point drop is not new information about your system; your system did not change. It is a measurement artifact: the synthetic angry cases, generated from complete specs with explicit hard constraints, are systematically harder than the real angry cases in your logs, which were often vague and quickly resolved. Admitting them into the headline score would make every future release look like a regression.

The operational rule: never blend. Report the headline score on real items only, and report synthetic segments as their own rows with the count visible. When a synthetic segment and a real segment disagree, that disagreement is the finding.

Keep an explicit quarantine: if an item later gets a real-log twin, prefer the real item and retire the synthetic one. Track provenance on every golden so that a future reviewer can see where each item came from and which generation prompt version produced it. Without provenance, a synthetic set degrades into an unattributable pile within a few quarters, and nobody can tell which items were ever reviewed.

For agents specifically, add trajectory-level checks to the advisory sample: did the agent pick the right tool at the right step, and did it reach a correct answer through a defensible path. A correct final answer can hide broken reasoning, and that hidden failure is invisible to output-only scoring LangChain. An agent-as-a-judge setup, where the evaluator can inspect intermediate steps rather than only the final message, is the natural fit when your synthetic tasks describe multi-step behavior arXiv.

Synthetic tasks and LLM judges have the same failure mode: both are generated by a model and both can drift away from your domain. The defense is the same too. Build the judge against a benchmark set you have annotated by hand, measure the judge against that ground truth, and iterate the judge prompt until it clears your threshold DeepEval. MLflow's guidance is that judges should be aligned with human feedback so automated scores stay calibrated to your team's standards MLflow, and Evidently's guide covers the prompt and scoring choices that keep a judge from drifting, including chain-of-thought and reference-based scoring Evidently AI.

Calibrating a judge on synthetic angry-customer items is circular: the judge and the items share a generator's biases. Calibrate on your reviewed sample, which contains both real and verified synthetic items, and report agreement separately for each group. If the judge agrees with humans on 90 percent of real items and 70 percent of synthetic ones, either the judge or the synthetic set is wrong, and that gap is worth a week of work.

One more calibration habit: hold the judge to a stated threshold before you use it to admit items. A judge that agrees with humans on only three of four boundary cases is fine for monitoring trends and not fine for signing off on new golden items, so keep a fixed review backlog of disagreements and revisit it when the judge prompt changes Google Cloud.

For teams running evaluation across many model providers, a single endpoint for both the system under test and the judge reduces the chance that the judge's provider bias leaks into the score (https://quantaintelligence.ai/inference).