Logs, Events and Spans: Picking the Right Signal per Step

A span attribute is a key-value pair attached to a span. In OpenTelemetry, spans carry name, parent span ID, start and end timestamps, span context, attributes and events, which is why a span looks a lot like a structured log with hierarchy baked in (OpenTelemetry traces).

The decision rule: if you can imagine writing a GROUP BY or a dashboard filter on it, it is an attribute. That means the model, the operation name, the provider, input and output token counts, the tool name, the status and the duration. For agent and model work, the GenAI semantic conventions standardize these names under a single gen_ai.* namespace so that a chat span from one framework and a raw SDK call from another speak the same vocabulary: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens among them (OpenObserve).

That standardization is not cosmetic. If one engineer names the attribute model and another names it llm_model_name, neither composes into one dashboard, and you cannot answer "what is our error rate by model" without a translation layer (OpenObserve). The conventions themselves are still moving, and the GenAI attribute registry page now redirects to a dedicated repository while marking the older attributes deprecated (OpenTelemetry registry, semantic-conventions-genai). Pin the names you use in a schema file and upgrade deliberately.

Two cautions. First, attributes are for values with a single value per span; if something can happen more than once inside a step, a single-valued attribute cannot record all the occurrences (Honeycomb). Second, high-cardinality strings in attributes are expensive and often sensitive. Keep the raw query and the full document text out.

The four golden signals from Google's SRE book give you a checklist for which attributes to add: latency, traffic, errors and saturation, measured at the service boundary where the user lives (novaaiops, Google SRE). For an agent step, that translates to a latency attribute on every model and tool span, a status or error-class attribute, token counts as your traffic proxy, and queue depth or concurrency as saturation. Latency needs care: track the latency of successful requests separately from failed ones, because errors often fail fast and pull your average down, making a failing service look healthy (novaaiops). And measure percentiles, not averages. One worked illustration from real traffic shapes: 9,000 requests at 50 ms, 600 at 120 ms, 350 at 600 ms and 50 at 8,000 ms produce an average near 113 ms but a median of 50 ms and a p99 around 600 ms, so the average hides both the typical experience and the tail (OneUptime).

A span event is a named, timestamped record inside a span. In the trace model it sits in an events array on the span, alongside the attributes (OpenTelemetry traces). Use it for a moment that only means something in the context of the step: retrieval started, retrieval completed, the first token arrived, a retry began, a guardrail fired.

The rule that separates events from attributes is repetition. If a thing can happen more than once in a step, an attribute overwrites and an event accumulates. The Honeycomb guidance is explicit here: if something might happen more than once, a single-valued attribute cannot record them all, and a fixed-time event like an interrupt or an error is a better fit for an event than an attribute (Honeycomb).

You should know where the platform is heading before you build on this. OpenTelemetry is deprecating the Span Event API because two overlapping ways to emit events (span events via Span.AddEvent and log-based events) caused split guidance and duplicate concepts for operators. The direction is that events are logs with names, emitted through the Logs API and correlated to traces through context, while existing span event data and trace views keep working (OpenTelemetry).

The practical decision rule: emit the moment now, but route new code through your logging pipeline with trace and span ID attached so it is already in the shape the ecosystem is converging on. The deprecation concerns how new events are recorded, not the ability to see events attached to spans (OpenTelemetry).

Logs record discrete events and carry the granular detail about what happened and why, which is exactly what traces and metrics do not do: the specific error, exception or unexpected behavior inside a component (SigNoz). The rule: if a human will read it during a single-step debug, and no dashboard will ever group by it, it is a log line.

That covers the actual query text, the tool arguments as sent, the tool response body, the exception message and a truncated preview of the prompt. The critical requirement is correlation. A log line without a trace ID is a dead end, so make sure the log participates in the trace by including the trace and span ID (Honeycomb). The OpenTelemetry trace model supports this because context propagation is what lets spans and their attendant records be assembled into one trace regardless of where they are produced (OpenTelemetry traces).

There is a privacy dimension that most teams learn the hard way. Prompt and response bodies are not standard attributes by default in the GenAI conventions; they are separate, explicitly enabled events precisely because they routinely contain PII (OpenObserve). A workable pattern is to log a short preview plus a SHA-256 hash of the full prompt, and sample full prompt text at a small rate while always storing the hash and token counts. That lets you join traces to sampled prompt text without leaking sensitive data at full volume (techlusion).

import hashlib, json, time
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def emit_step_log(task_id, query, result_count, outcome):
    ctx = trace.get_current_span().get_span_context()
    print(json.dumps({
        "ts": int(time.time() * 1000),
        "trace_id": format(ctx.trace_id, "032x"),
        "span_id": format(ctx.span_id, "016x"),
        "task_id": task_id,
        "event": "retrieval_completed",
        "query_preview": query[:200],
        "query_hash": hashlib.sha256(query.encode()).hexdigest(),
        "result_count": result_count,
        "outcome": outcome,
    }))

The two fields that make this worth writing are the trace ID and the hash. The ID joins the line to the span timeline; the hash lets you count distinct queries and detect repeats without storing raw text.

Take a single agent step: the agent receives a question, calls a vector retriever, calls a "get_order_status" tool, then produces an answer. Here is how to split the data.

On the parent step span, as attributes: gen_ai.operation.name set to the step type, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, step.status as success or error, and step.duration_ms. These are the values you will group by across thousands of steps. Token counts are attributes because billing is per token, not per request, which is one reason LLM calls needed their own conventions in the first place (OpenObserve).

On child spans, as attributes: a retrieval child span with db.collection.name for the index, the top_k requested, and the number of results returned. A tool child span with the tool name, the upstream status code and the tool latency. The database span conventions specify that span names follow a {db.operation.name} {target} shape and that db.collection.name identifies the entity operated on, so use those rather than inventing index_name (OpenTelemetry DB conventions). Vector databases behave differently from relational ones because they run approximate nearest neighbor searches, so query latency depends on index size and dimensionality and can jump from tens of milliseconds to hundreds under load, which is why per-call latency on the retrieval span is worth having as an attribute (OneUptime).

As events on the step span: retrieval.started, retrieval.completed, rerank.completed, tool.started, tool.completed. Each carries a timestamp and maybe a score. If the retriever is called twice because the first pass returned nothing, you get two retrieval.completed events instead of an overwritten attribute.

As log lines: the actual query text, the IDs of the returned chunks, the tool arguments, the tool error message if it failed, and a 200-character preview of the retrieved context. All with trace and span ID.

In a separate event store: one row per step outcome with the trace ID, the task ID, a boolean for whether retrieval returned zero usable results, whether a safety check fired and which policy, and the token cost. Retrieval misses are the clearest case. You will want to analyze them over a month to find queries your index cannot answer, and traces will not be retained that long.

A reranker belongs in this picture because the two-stage pattern (retrieve broadly, then rerank down to the few documents you actually pass to the model) is what keeps retrieval recall high while keeping the context window small, since LLM recall degrades as you stuff more tokens into the context (Pinecone). That means the retrieval span needs two attributes that tell you whether the rerank stage did its job: documents retrieved and documents kept. Without them you cannot answer whether a bad answer came from a retrieval miss or a reranking mistake.

The cost structure differs by signal, and it differs by billing model within the signal. Providers that charge per event make adding attributes nearly free, since the marginal cost is network and long-term storage rather than a new billable record (Honeycomb). Providers that charge per byte treat a fat attribute and a log body the same way, so check which one you are on before you decide that attributes are free.

Three rules of thumb keep the bill predictable.

First, sampling differs by signal. Full metadata for every model and tool call, with prompt text sampled at a low percentage and hashes and token counts always stored, is a reasonable default (techlusion). Attributes survive sampling decisions better than bodies, because you can compute statistics from a sampled set of spans as long as the sample is unbiased.

Second, retention differs by signal. Traces and span events typically live on the trace retention clock, log lines on the log clock, and a separate event store on whatever clock your business needs. Anything you need for a monthly review of retrieval quality or a billing reconciliation must not live only in traces, or you will be querying an empty store on day 31.

Third, put the SLO on the outcome, not the internals. SLOs for agents should be user centered: task success rate, end-to-end latency at a high percentile, and a hallucination rate measured on a verified sample, with LLM provider latency and tool failure rate as secondary indicators (techlusion). This is the same logic as classic reliability work, where the SLI is the ratio of good events to total events and the error budget is one minus the SLO, a decision tool rather than a penalty (OpenObserve, Srikanta Sahu). Set the internal SLO tighter than any external promise so alerts fire before a contractual breach (OpenObserve).

If you are wiring many models into one agent, the plumbing that emits these attributes is often more work than the observability backend. Quanta Inference puts text, image, video, audio and embedding models behind one OpenAI-compatible API key, which keeps the gen_ai.request.model attribute meaningful across providers instead of fragmenting per vendor (Quanta Inference).

Use this before you ship a new agent step.

One more habit worth forming: name spans after the work, not the library. The span conventions call for a span name that reflects the operation, and a database span should be named from the query summary or operation and target rather than the client class (OpenTelemetry DB conventions). A trace reads as a story only if each chapter has a name that describes the plot.