A Synthetic Task Is a Contract: Fields a Generator Must Fill

A working spec needs exactly six groups, no more as mandatory fields and no fewer. Each maps to a question a grader would otherwise have to answer alone.

The intent field deserves special emphasis. Google's framework insists on a measurable success statement per agent, contrasting "the agent should be helpful" with "success is a factually correct, concise summary fully grounded in known documents" Google Cloud. A synthetic task with a vague intent inherits the same defect, and your rubric inherits it too.

Intent should be a short string plus a pointer to a capability registry, not a paragraph. The registry is a list of capabilities your product actually claims, such as "extract line items from an invoice" or "refuse an out-of-policy refund request." One task, one entry.

A worked example makes the cost concrete. Suppose a generator emits a task whose intent reads "handle a refund request correctly." The grader then has to decide whether correctness means policy compliance, tone, identity verification, or the refund actually being processed. A support agent handling a refund across eight conversational turns makes each of those a separate opportunity for failure LangChain. If the task records one intent, the rubric can check one thing and a pass is interpretable.

The decision rule: if you cannot name the capability in a noun phrase that also appears in your product documentation, the task is not ready. Split it, or drop it.

The source slice is where most generated-task pipelines quietly break. A source slice should contain three things: a stable identifier for the source record (a primary key, not a row number), a content checksum over the exact bytes the task used, and the path or query that produced it.

Why both ID and checksum? The ID lets you find the record again when you investigate a failure. The checksum lets you detect that the record changed under you. Schema-inference research shows that document stores permit structural drift and partial documentation, so a task built from a record that later mutated will grade against an input nobody can reconstruct the SVEF framework paper on schema validation. A checksum turns that silent mismatch into a loud lint failure.

The same principle appears in golden-dataset practice: a curated, trusted set is used for benchmarking rather than learning, and its value comes from being stable enough that a score movement means something DAC.digital on golden datasets. If your synthetic task cannot name its source slice precisely, it is not a fixture, it is a rumor.

Expected behavior is a description of a correct response, stated as observable properties. "Returns JSON with the fields invoice_id, total, and currency, where total equals the sum of line items" is a spec. "Answers well" is not. Rubric-writing guidance makes the same point: the key attribute of a criterion is that it is observable, with clear definitions of success and edge cases Twine's guide to writing an LLM evaluation rubric.

Allowed tools is the field teams skip most often, and it is the one that catches deceptive passes. List every permitted call explicitly. Anything not on the list is a hard failure even if the final answer is correct. Google's framework places tool selection accuracy inside its process-quality pillar precisely because output-only scoring cannot see it Google Cloud. LangChain reports the same: an agent that hallucinates a tool call and still gets lucky looks fine to an output-only grader LangChain.

A practical rule: if a task needs more than three tools, split it, or record the permitted sequence. Trajectory evaluation exists because the path matters, and a tool allowlist is the cheapest way to make the path checkable the agent evaluation crash course.

An empty failure rubric is worse than no rubric, because it silently converts the grader into a vibes check. The rubric should list named failure modes, each with a short definition and, where possible, a deterministic check.

For each task, write at least one failure mode with an exact condition. For example:

failure_modes:
  - id: missing_currency
    when: output.total present and output.currency absent
    check: code
  - id: wrong_policy_citation
    when: agent cites a refund window other than the one in source slice
    check: llm_judge
  - id: out_of_scope_tool
    when: any tool outside allowed_tools appears in trace
    check: code

Judges that score against structured criteria are more consistent than free-form judges, and combining code checks with an LLM judge covers both deterministic and subjective failures Confident AI's guide to LLM-as-a-judge. Google's comparison of evaluation methods notes that code-based checks are the cheapest and most deterministic, while judges are best for subjective qualities at scale Google Cloud. Writing both kinds into the rubric, labeled by check type, is what lets a pipeline run cheap checks first and reserve judge calls for the cases that need them.

Provenance is the field that costs nothing to write and saves an afternoon when a batch goes bad. Record the generator identity, the generation parameters that matter, the source slice checksum, and the spec version. When a batch starts failing a regression test, provenance is what lets you retire exactly the affected tasks instead of quarantining everything.

Provenance also supports reuse. A randomized, deterministic pipeline that samples tasks from a golden set should record enough to regenerate any given task, which is what makes a benchmark worth trusting over time DAC.digital. The ChatLog work shows the value of this discipline at the research scale: large-scale records of model responses kept over a long period let researchers detect step-wise changes that a one-shot benchmark would miss the ChatLog paper. Synthetic task sets without provenance cannot support that kind of longitudinal analysis.

The check is a single function with three reject conditions: any required field missing or empty, an empty failure rubric, or a source slice that does not resolve to a real input with a matching checksum. Reject means the task never enters the eval set. Do not auto-fill, do not flag for later.

REQUIRED = ["intent", "source_slice", "expected_behavior",
            "allowed_tools", "failure_rubric", "provenance"]

def lint_task(task, resolver):
    errors = []
    for field in REQUIRED:
        if not task.get(field):
            errors.append(f"missing:{field}")
    rubric = task.get("failure_rubric") or []
    if not rubric:
        errors.append("empty:rubric")
    elif any(not m.get("when") for m in rubric):
        errors.append("incomplete:rubric")
    slice_ = task.get("source_slice") or {}
    record = resolver.fetch(slice_.get("id"))
    if record is None:
        errors.append("unresolved:source")
    elif checksum(record) != slice_.get("checksum"):
        errors.append("stale:source")
    if not task.get("allowed_tools"):
        errors.append("missing:tools")
    return errors

Run it on every generated task, before human review and before any model call. Two operational details matter. First, run the lint in the generator's own CI, so a bad template fails fast rather than producing a thousand broken tasks. Second, sample the rejected tasks periodically; a rising rejection rate usually signals a source system drifting, not a generator bug.

Teams building synthetic eval sets across many models and modalities have a second problem after lint passes: running the grader cheaply and consistently. Routing judge calls through Quanta Inference keeps the pipeline behind one OpenAI-compatible key, which matters when the grader compares responses from several providers and you want costs in one prepaid balance.