Feeding a Failed Sample Back Into the Tier's Prompt or Tool

The first rule of the loop is that you do not rewrite the prompt until you can replay the failure. A trace records the execution path of a single run as connected spans, each with its inputs, outputs, timing, tool calls, retrievals, model calls and state changes per Braintrust. That structure is what lets you say "the retrieval span returned document X, and the model then cited it" rather than "the model hallucinated."

There is a useful distinction between three artifact types, and conflating them is where most loops stall as Braintrust sets out:

For agent work you need the third. A single run typically involves multiple model calls, tool calls, retrievals and state changes, and any one of them can introduce the error while the outer request still reports success per Braintrust. Several tracing stacks now emit spans in OpenTelemetry conventions, which matters if you already run an observability backend, and agent platforms are adding native span export; the OpenInference conventions described in Arize's walkthrough are one example of that portability.

Practical rule: if a failure cannot be reproduced from the captured trace alone, you do not yet have enough instrumentation. Rerunning locally and adding print statements is a sign the trace is missing a span, not a sign the model is erratic.

The classification step is where discipline pays. The failure label determines which part of the contract you are allowed to change, and choosing the wrong layer is the most common cause of a fix that does not hold. A rough map that works well in practice:

The distinction between a prompt fix and a schema fix is worth its own paragraph, because teams reach for the prompt far too often. If the model calls a refund tool with a negative amount, no amount of "never pass negative values" in the system prompt is as reliable as a schema constraint that rejects the call before it executes. Language models respond to instructions probabilistically; validators respond deterministically. Put the constraint where enforcement is guaranteed, and reserve prompt clauses for things a validator cannot express, such as tone or the decision to ask a clarifying question.

The same logic applies to side effects. Microsoft's agent framework lets you wrap a function tool in an approval-required wrapper, so a run that needs a human decision completes with a ToolApprovalRequestContent request instead of executing the function, and the caller passes the approval back in a new run per Microsoft Learn. That is a contract change at the orchestration layer, and it is the right fix when the failure label is "unsafe action attempted."

Narrowest means: the smallest edit that, applied to the old trace, would have converted the failure into a correct or safely-halted run, with no other behavior changed. Test that counterfactual mentally or, better, by replaying the trace with the change applied. If a clause you are adding would also alter ten passing runs, it is too broad.

A worked example with numbers makes this concrete. Suppose your evaluation set is 400 samples and 46 fail, a pass rate of 88.5 percent. You sort the 46 by label and find 21 are invalid tool arguments (mostly a missing currency field on a payment tool), 14 are retrieval misses on a policy question, 7 are format violations, and 4 are unsafe actions. Now the decision is easy:

• The 21 argument errors get a schema fix: currency becomes a required enum, and the tool rejects the call. Expected recovery if the schema was the cause: close to all 21.
• The 14 retrieval misses get a query and filter fix plus a relevance check. Recovery is uncertain, so you re-run those 14 as a small batch and measure.
• The 7 format violations get one output validator with a retry, which converts a failure into a bounded retry rather than a broken response.
• The 4 unsafe actions get an approval gate, which does not make them pass; it makes them wait for a human.

Notice that the fanciest change, a rewritten system prompt, is not on the list. That is the insight most first-page advice misses: the prompt is the easiest thing to edit and therefore the most tempting, but prompt edits are the least testable and the easiest to over-apply. Spend your effort on the deterministic layers first and return to the prompt only for failures that genuinely are about instruction, such as tone or when to ask a question.

Reversibility should also guide the fix. If your agent has approval gates, decide per action type what happens when nobody answers, because a queue without a timeout policy fills with items nobody decided per Orbiresearch's write-up on approval timeouts. Their recommended defaults are chosen by cost of undoing the action: approve for internal, reversible, low-impact actions; hold and escalate for anything that leaves your system or moves money; cancel for actions whose value expires; re-plan for actions built on data that may be stale. That is a contract decision you make once per action type, in writing, and it converts a class of failures into a defined behavior instead of an outage.

Durable approval waits are a solved pattern if you use a workflow engine. Temporal's approval pattern blocks on a condition with a timeout and unblocks on a signal carrying structured data (approver, decision, comments, timestamp), which also gives you an audit record for free per Temporal's docs. Cloudflare's agents documentation shows the same shape with a durable waitForApproval() gate that can hold for months without keeping an agent running per Cloudflare's docs. Either way, the approval is a contract element, not a chat message.

A fix that is not pinned will come back. The mechanism is straightforward: someone edits the prompt for an unrelated complaint, the edit weakly conflicts with your constraint, and the old failure returns through a new input. Regression testing exists precisely because a prompt tweak that fixes one complaint can quietly break other answers per Langfuse's regression testing guide.

The fixture format matters. A production trace should become a dataset row that preserves the original input, the expected behavior where a gold answer exists, and metadata including the failure label, source span, timestamp, prompt version and model version per Braintrust's five-step workflow. Keeping the prompt and model version is what lets you tell a real regression from a harness drift when the fixture fails six weeks later.

Here is a minimal fixture in the shape most eval harnesses accept. The exact schema is yours; the fields are the point.

{
  "id": "refund-currency-001",
  "failure_label": "tool_argument_error",
  "source": "prod-trace-8f21",
  "input": "Refund order 5512 for 42 back to the card on file.",
  "expect": {
    "tool_called": "issue_refund",
    "arguments_valid": true,
    "no_missing_required_fields": true
  },
  "must_not": ["issue_refund with currency null"],
  "contract_version": "refund-tool-schema-v2",
  "scorer": "schema_validity"
}

Two properties make a fixture useful. First, it must fail on the old contract. Before you merge the schema change, run the fixture against the previous configuration and confirm a red result; if it passes there too, your test is checking the wrong thing. Second, it must assert on the step that failed, not only the final text. A scorer for invalid tool arguments should inspect the tool-call span, not ask a judge model whether the answer "looks right," because the answer may look fine while the argument was wrong per Braintrust's guidance on scorers.

Gate releases on the whole fixture set. Langfuse frames this as failing CI before regressions ship per Langfuse, and the same scorers can run as online scoring rules on live traffic, so the pattern you diagnosed in production keeps being checked after deployment per Braintrust.

If your fix is an approval gate, you have moved the failure mode from "the agent does the wrong thing" to "the human rubber-stamps the agent." That is a real risk with measured numbers behind it. Claude Code users approve 93 percent of permission prompts, and in a study Anthropic ran on 1,053 paid testers, human review caught 13.6 percent of harmful actions while its automated classifier caught 89 percent as reported by Beri. In a preprint by Ting Yan, reviewers approving each action let through 40.4 percent of attempted overreach, and that was the best of the three designs tested as summarized by RockCyber Musings. The rock bottom line: a gate is only part of the contract if someone reads it.

Two consequences for the loop. First, gate far fewer actions: reserve approval for irreversible or externally visible steps, and let a schema or policy handle the rest. Second, when a gate failure appears in a trace, the fixture should assert on the gate itself, for example that an action of type issue_refund above a threshold was routed to a human rather than auto-approved. Reviewers also degrade with volume and with confident rationales attached to proposals, a pattern documented as automation bias; a Harvard Business School study with 228 evaluators found reviewers were 19 percentage points more likely to align with AI recommendations than a control group, and narrative rationales increased deference by another 5 points per Tianpan's summary. Design gates that are rare enough to be read.

Put the whole thing together on one case. The trace shows a customer asking for a refund, the agent calling issue_refund with a missing currency, the tool accepting it, and the final answer being a confident confirmation. Four steps follow.

Step one, capture. Store the trace with its spans, prompt version and model version.

Step two, classify. Label it tool_argument_error. Do not label it "bad answer," because that label leads to the prompt.

Step three, change the contract. Make currency a required enum in the tool schema, and add a validator that rejects the call before execution. One line of schema, no prompt edit.

Step four, pin it. Add refund-currency-001 to the fixture set, verify it fails on the prior schema version and passes on the new one, and add it to the CI gate.

Now count the economics honestly. One diagnosed failure becomes one fixture. Your 46 failures become perhaps 30 fixtures after deduplication, because several samples usually share a root cause. Those 30 fixtures run in CI on every prompt, tool and model change. The next time someone edits the refund flow, the fixture catches the regression before it reaches a customer. That is the compounding return the pass rate alone never gives you: a growing, specific test suite that encodes everything your team has learned about this agent.

Sampling the tail is the honest way to keep coverage growing without reading every trace. Auditing has the same problem and a mature answer: define tolerable deviation, choose items by risk, and evaluate the sample against a stated confidence level rather than intuitively per the PCAOB's audit sampling standard. You do not need audit formality, but the habit translates: review a fixed number of traces per period, prioritize the ones with side effects or low online scores, and log what you found as labels.

For teams running agents across several model providers, the trace and fixture plumbing is the part that benefits most from a single interface. Quanta Inference gives you text, image, video, audio and embedding models through one OpenAI-compatible API key, which keeps the fixture harness stable when you swap the model behind a failing span. If your loop is blocked because nobody owns the harness, an expert engagement can set the trace, label and CI gate up with your team.