A Trace for a Multi-Agent Run: What to Record Across Hops

The rule is simple. Generate one trace_id when the parent task starts. Every step in that run, including the orchestrator's own reasoning, each subagent call, each tool call fetched by a subagent, and the final merge, gets a fresh span_id and a parent_span_id pointing to the span that caused it.

The W3C Trace Context specification standardizes the headers and value format for this, and the traceparent field has four parts: version, trace-id, parent-id and trace-flags the W3C Trace Context recommendation. OpenTelemetry's default propagator uses those W3C headers, and its context propagation model is what makes a trace causal across process and network boundaries OpenTelemetry context propagation.

For agents this matters because the W3C model assumes a synchronous request that carries a header. Agents break that assumption in three ways: they hand off work asynchronously through queues, they call external tool servers that do not automatically receive headers, and framework internals run tool loops that expose no hooks TianPan on tracing across agent boundaries. The fix is explicit injection and extraction: capture the active span context before crossing any boundary, serialize it into the message payload or metadata, and reattach it on the receiving side before creating new spans.

The A2A project has an open proposal for exactly this, carrying traceparent and tracestate in message metadata and reusing existing OTel GenAI attribute names such as gen_ai.agent.name and gen_ai.operation.name, plus three A2A-specific attributes (a2a.task.id, a2a.task.state, a2a.method) the A2A tracing proposal. The proposal also notes that everyone currently invents their own attribute names, which is why dashboards break when you switch tools. Adopt the OTel GenAI names where they exist and add only what your protocol needs.

A minimal header set to propagate on every hop:

traceparent: 00-<32 hex trace_id>-<16 hex span_id>-01
tracestate: tenant=<tenant_id>,workflow=<workflow_version>
x-quanta-task-id: <parent task id>
x-quanta-agent-name: <target agent name>

Keep baggage out of the default path unless you need it. The A2A proposal specifically excludes baggage from its first version because baggage propagates in plaintext to every downstream service, which is a data leakage vector in multi-tenant deployments the A2A tracing proposal.

The parent task span is the root of the tree and sets the run's identity. Record it once, before any delegation, and write it even if the run crashes early.

Log the goal text and not just the hash, because an input hash proves identity but tells a reviewer nothing. Log the budget separately from the spent amount so you can tell a graceful stop from a crash.

Each delegation creates a child span under the orchestrator span that made the decision. This is the hop where most detail belongs, because a worker is the unit you will replay.

Two fields earn their place more than the rest. delegation_reason is what lets a reviewer ask why the orchestrator sent a task to the billing agent at all, which is the first question in a misrouting failure. The A2A proposal points out that orchestrators can misclassify a task and that context overflow is common once four or more workers accumulate output, so a written reason per delegation is the cheapest guard against both the A2A tracing proposal. stop_reason separated from error_code lets you distinguish a clean but truncated result from a hard failure, and truncated results are a frequent cause of confidently wrong merged answers.

Tool inputs and outputs are the riskiest fields to log because they carry user data, credentials, and the contents of retrieved documents. Log a hash and a reference, and store the full payload in a separate store with its own retention and access rules. A hash gives you equality checks, and a reference gives you a lookup when someone with permission needs to see the text.

The merge span is a child of the parent task and a sibling of the subagent spans. It records what the orchestrator chose, not what the workers produced.

• selected_output_refs: the list of span IDs whose outputs went into the final answer
• excluded_output_refs and exclusion_reason: which worker results were dropped and why
• conflicts: pairs of worker results that disagreed, with a short description of the disagreement
• output_hash and output_ref: the exact final text that was returned to the user
• merge_prompt_id, merge_model_id, merge_model_params: the synthesis step is its own model call and needs its own configuration
• stop_reason: why the run ended, for example orchestration complete, token budget exhausted, human override, or error escalation
• human_review: whether a person approved, edited, or rejected the result
• tokens_in, tokens_out: merge tokens, which grow with the number of workers

Conflicts deserve a structured field, not a sentence inside a free-text summary. The Beam AI analysis of orchestration patterns notes that when parallel agents disagree the system needs an explicit conflict resolution strategy rather than a generic summarization step, and that LLM-based synthesis can invent a consensus that does not exist in the underlying data Beam AI on orchestration patterns. Recording the conflicting pair and the resolution rule turns a silent wrong answer into a queryable event.

Context propagation has two halves: context is the object holding the trace and span identifiers, and propagation is the mechanism that serializes it and moves it across a boundary OpenTelemetry context propagation. For agent systems, the practical rule is to serialize the context yourself at every point that is not a plain HTTP request.

from opentelemetry import trace
from opentelemetry.propagate import inject, extract

tracer = trace.get_tracer("multi_agent")

def delegate_to_subagent(task, payload, queue):
    with tracer.start_as_current_span("invoke_agent " + task["agent_name"]) as span:
        span.set_attribute("gen_ai.agent.name", task["agent_name"])
        span.set_attribute("a2a.task.id", task["task_id"])
        span.set_attribute("delegation.reason", task["reason"])
        carrier = {}
        inject(carrier)              # writes traceparent from the active span
        message = {"task": task, "payload": payload, "trace": carrier}
        queue.put(message)

def worker_loop(queue):
    while True:
        message = queue.get()
        ctx = extract(message["trace"])   # reattach before creating spans
        with tracer.start_as_current_span("process_message", context=ctx) as span:
            result = run_task(message["task"], message["payload"])
            span.set_attribute("stop.reason", result["stop_reason"])
            span.set_attribute("output.hash", result["output_hash"])

The inject call writes the current traceparent into the carrier dictionary, and extract rebuilds the parent context on the receiving side so the new span attaches under it instead of starting a fresh trace. Because a carrier is just a dictionary, the same pattern works for message queues, database rows, or Redis keys TianPan on tracing across agent boundaries.

For agent-to-agent traffic, the W3C format itself is short enough to carry in metadata: version, trace-id, parent-id, trace-flags, as in the example 00-a0892f3577b34da6a3ce929d0e0e4736-f03067aa0ba902b7-01 OpenTelemetry context propagation. Record the full traceparent string on the receiving span as an attribute as well as in headers, so you can still join traces when a message is replayed from a dead-letter queue with headers stripped.

A trace that contains every prompt is a privacy liability. Three rules cover most cases.

First, never log raw credentials, API keys, or full request headers. Redact at the logging call, not at the storage layer, so a mistake in one sink cannot leak. Second, treat full prompt and tool output text as a separate store keyed by hash, with its own retention window and access control, because multi-tenant systems are where cross-tenant leakage actually happens. Third, keep the trace record itself long enough to replay a regression and short enough to satisfy your data policy: identity, hashes, timings and error codes age well, while raw text does not.

A workable split is to keep the index (trace_id, span_id, parent_span_id, hashes, model IDs, timings, tokens, error codes, stop reasons) for the full period you support incident review, and keep payload text for a much shorter window. When you redact a payload, record that the redaction happened, so a later reader knows the missing text is policy and not a logging bug.

Replay is the payoff of the schema, and it works because a hash is a stable pointer. To reconstruct and re-run a failed hop:

1. Start from the trace_id and pull all spans ordered by parent_span_id.
2. Find the first span where stop_reason or error_code departs from a normal completion.
3. Fetch the stored text behind that span's task_description_hash and its prompt_id and model_id.
4. Check the model_params before re-running, because a changed temperature produces a different answer and invalidates the comparison.
5. Re-run only that span with the original context, then compare output_hash against the recorded value and diff the merged result.

Replay against a fixed model configuration is a fair test of a prompt change. Replay against a different model configuration is an experiment, not a regression check, so label it that way in the run record. Keep both hashes in the new span so the comparison is auditable.

This is also where the architecture choice matters. Multi-agent systems earn their cost on tasks that decompose into independent parallel threads, and are a poor fit for tightly interdependent work where workers need to see each other's decisions Anthropic's multi-agent research system. If your orchestrator decomposes badly, no trace will fix the design. The trace will, however, show you the bad split quickly, because delegation_reason and excluded_output_refs sit next to each other in the record.

When runs fan out across many models and providers, the per-hop model ID becomes a routing signal as well as a debugging field. Teams that run several families of models through one endpoint can capture a single model_id string per hop and keep the rest of the schema unchanged; Quanta Inference (https://quantaintelligence.ai/inference) is one way to put text, image, video, audio and embedding models behind a single OpenAI-compatible API key, which keeps the identifier recorded at each hop consistent.