Span Attributes That Let You Slice by Model, Tenant and Outcome

A workable default set for a RAG or agent service looks like the table below. The right column explains what each one unlocks.

Two details are worth more than the table.

First, the model attribute. OpenTelemetry's GenAI conventions name gen_ai.request.model and gen_ai.response.model as the standard fields for this (GenAI semantic conventions guide). Emitting the provider's full model string is tempting because it is free, but provider strings tend to include dated snapshots and deployment suffixes, which multiplies the distinct values. Decide deliberately: emit the full string on the span for forensics, and derive a short family value for dashboards, either at collection time or in the query. Pick one place to do that derivation and document it, or two teams will compute two different families.

Second, the outcome. Do not rely on span status alone. A span status of error tells you something failed; it does not tell you whether the failure was your prompt, the provider, or a quota. Normalizing into a small closed set is what lets you build burn-rate alerts on a ratio of good events to total events, the standard shape of a service level indicator (how to set meaningful SLOs). Free-text error messages make that ratio impossible to compute, because you cannot group by a sentence.

A practical naming rule: pick a prefix and keep it. tenant.*, retrieval.*, and outcome.* are self-describing, filterable as groups, and unlikely to collide with vendor instrumentation. The GenAI conventions themselves use a single gen_ai.* namespace for exactly this reason, and having every library agree on the same names is the whole point of semantic conventions (semantic-conventions-genai repository).

Cardinality is not a vague warning. It is arithmetic you can do before you ship.

Suppose you emit a user identifier on every span. Your service handles a large volume of requests a day across a wide user base. Every distinct value of that attribute is a separate series in the backend's index, and that index usually lives in memory for fast filtering. On the logging side, the same lesson shows up as the difference between a field called user_id that you can group by and a sentence containing a user number that you cannot (structured logging in production). Spans aggregate the same way, so high-cardinality attributes are the fastest route to a backend bill that grows faster than traffic.

Compare the attributes below. Same service, same traffic, wildly different index cost.

The pattern is that cardinality tracks the number of distinct logical entities, not the number of requests. Tenant identifiers are the tricky case: the count is bounded, but it can still be large enough to make a grouping over all tenants unusable. A tier attribute answers the one query you actually run at 3 a.m. ("is enterprise slow?"), and a hashed, truncated tenant key answers the follow-up ("is this one customer slow?") without exposing the identifier itself.

Two things to check whenever someone proposes a new attribute. What is the expected number of distinct values over a month? And is that attribute a function of something already on the span? If a tenant.plan_name can be derived from tenant.tier, you do not need both.

The clean mental model is a two-tier one. The span carries the attributes that need to be indexed and grouped. The detail that only matters for one request goes somewhere else, joined back by trace ID or span ID.

OpenTelemetry supports that natively. A span has attributes, events and a span context, and the trace carries a shared identifier across services (traces concept). Span events are the right home for the prompt that produced a bad answer or the retrieval scores behind a marginal hit, because they do not need to be filtered or grouped.

The GenAI conventions make a related decision for you. Prompt and response bodies are not standard attributes; they are separate, explicitly enabled events, because they routinely contain personal data (GenAI conventions guide). That is a good default to copy even if you never touch a GenAI-specific library: content that can contain identifiers, contracts or health details should be opt-in and stored apart from the queryable index. The conventions registry documents which attribute names exist and which are already deprecated in favor of the dedicated repository, so check names there before you invent one (Gen AI attribute registry).

For the log side, structured logging gives you a consistent shape for the detail. A JSON line with trace_id, user_id and the raw error string is searchable during an investigation and cheap to retain on a different schedule than your span index (structured logging: what it is and why you need it). The join key is the trace ID, which OpenTelemetry context propagation carries across service boundaries automatically (OpenTelemetry overview).

The decision rule, then: if you want to group, filter, or compare, it goes on the span. If you want to read it during a postmortem, it goes in a log line or a span event with the trace ID attached. If it is both, put a normalized version on the span and the raw version in the log.

One more consideration: timestamps. Structured logs should carry an ISO 8601 UTC timestamp and a severity level, since that is what makes ordering and time-window queries reliable across systems (structured logging guide). Spans already carry start and end times, so the log line is what you correlate against them, not what replaces them.

Consider a concrete case with numbers, so the arithmetic is visible rather than asserted.

A retrieval-augmented assistant serves two million requests a day. Roughly 1,900,000 are chat, 100,000 are embedding calls. You notice the 99th percentile for chat is climbing while the median stays flat. Averages would hide this, which is the standard objection to mean latency: a handful of slow outliers drag the mean without describing the typical user. Google's SRE guidance on monitoring makes the same argument at the signal level, recommending that you measure distributions rather than averages, and that you track the latency of successful requests separately from failed ones, because fast failures can make a sick service look quicker than a healthy one (monitoring distributed systems). That is exactly why outcome.class belongs on the span.

With the core attribute set in place, the investigation is a sequence of groupings rather than a hunt:

1. Group by outcome.class. Say 96 percent of chat spans are success, 2 percent are provider errors, 1.5 percent are rate limited, and the rest are client errors. The rate-limited slice is a quota conversation with your provider, not a code bug.
2. Filter to success and group by retrieval.outcome. Suppose hit spans run a p95 of 900 milliseconds and miss spans run 1,400 milliseconds. The gap tells you the fallback path (no context, longer prompt, more output tokens) is the slow branch. That is a design decision, not an infrastructure problem.
3. Group the miss branch by tenant.tier. If enterprise tenants dominate, their corpora are probably larger or their filters tighter, and you have a specific tuning task.
4. Pull the trace IDs for the twenty slowest spans and read the span events and logs. This is where the full prompt and the retrieval scores live, and this is the only step that needs them.

The point of the example is that steps one through three never touch prompt text. They are pure aggregation over five or six low-cardinality attributes. Only step four is forensic, and it operates on twenty traces, not two million.

A second example shows how to keep per-tenant visibility without unbounded cardinality. If your top accounts generate most of the revenue and most of the load, put the hashed tenant key on their spans and leave it off for everyone else, or emit a tenant.tier value of long tail for the rest. You get per-account p95 where it matters, and a single bucket where it does not. The four golden signals, latency, traffic, errors and saturation, give you the vocabulary for those per-bucket views, and none of them require a user identifier (the four golden signals).

Run new attributes through this list. It is short enough to paste into a pull request template.

• Named query. Which dashboard or alert consumes this attribute? If none, do not add it.
• Distinct value estimate. What is the count over a month? Anything unbounded or proportional to users goes elsewhere.
• Sensitive data. Does the value contain a name, email, account number, address, or text a customer typed? If yes, it goes to an opt-in event or a log, not the span.
• Derivable. Can it be computed from a sibling attribute at query time? Then skip it and document the derivation.
• Enumerated. Is the value set closed and documented, or is it free text? Closed sets can be grouped and alerted on; free text cannot.
• Stable name. Does it follow your prefix convention, and is it the same name in every service that emits it? Inconsistent field names across services are the single most common reason a cross-service query fails, as structured logging practitioners consistently report (structured logging guide).
• Tied to an SLI. Does at least one value feed a ratio of good events to total events? Those ratios are what error budgets are built on (SLOs, SLIs and error budgets).

Retrieval deserves one extra note. If you run a two-stage pipeline with a vector search and a reranker, the retrieval outcome attribute should describe the whole retrieval step, not just the vector call. Reranking exists precisely because vector search alone misses relevant documents, and it costs latency and money (rerankers and two-stage retrieval). A retrieval.outcome of hit after reranking means something different from a hit before it, so record the stage. Vector search latency itself is worth measuring as its own span, since query shape, filters and index size all move it around (monitoring vector databases with OpenTelemetry).

Two implementation habits keep the attribute set from drifting.

The first is a small helper that every LLM call goes through, so the core attributes are set in one place. Something like this, in outline:

def llm_span(tracer, operation, model_family, tenant, retrieval):
    span = tracer.start_span("llm.call")
    span.set_attribute("gen_ai.operation.name", operation)
    span.set_attribute("gen_ai.request.model", model_family)
    span.set_attribute("tenant.tier", tenant.tier)
    span.set_attribute("retrieval.outcome", retrieval.outcome)
    return span

# outcome.class and tenant.key_hash are set in the same wrapper on exit,
# once the result or exception is known.

The second is a review step in the collector or the SDK that can drop or rename attributes, so a well-meaning library cannot quietly add a user identifier to every span. OpenTelemetry's collector is built for exactly this: it processes and routes telemetry before it reaches any backend (OpenTelemetry overview), and vendor neutrality means you can change backends without re-instrumenting.

If your spans already carry the right attributes, the queries are a few lines wherever you store them. The shape is always the same: filter to a population, group by one low-cardinality key, compute a percentile on duration.

SELECT
  gen_ai_request_model AS model,
  tenant_tier,
  retrieval_outcome,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms,
  count(*) AS spans
FROM spans
WHERE gen_ai_operation_name = 'chat'
  AND outcome_class = 'success'
  AND start_time > now() - interval '1 hour'
GROUP BY model, tenant_tier, retrieval_outcome
ORDER BY p95_ms DESC;

That single query replaces the three-dashboard scavenger hunt that most teams run today. It also makes the cost question visible: the cardinality you pay for is the number of rows this query groups over, and every attribute you did not add is money you did not spend. Be aware that agent and tool-orchestration conventions are still moving faster than the core chat attributes, so keep the parts you depend on narrow and versioned (GenAI conventions guide). If your service calls many providers and you would rather not maintain a separate SDK path for each, routing them through one OpenAI-compatible endpoint keeps the instrumentation to a single client (Quanta Inference).