Tracing the Query Path: Spans for Retrieval, Reranking, Calls

Draw the tree first. A workable default for a single RAG request looks like this:

rag.request                    (parent, INTERNAL or SERVER)
├── embed.query                (CLIENT, to embedding model)
├── vector.search              (CLIENT, to vector DB)
├── rerank                     (CLIENT, to reranker)
├── prompt.assemble            (INTERNAL)
└── model.generate             (CLIENT, to LLM provider)

Three design rules make this tree useful rather than decorative.

Rule 1: one span per outbound dependency, plus one per stage you own. The four CLIENT spans map to four networks you do not control. prompt.assemble is the INTERNAL span for work you do control (template rendering, token counting, truncation). If prompt assembly is slow, no vendor is at fault.

Rule 2: the parent span should be INTERNAL unless it is genuinely serving a request. The span kind is a hint about what the span represents, and database client spans, for example, are specified to be CLIENT and may fall back to INTERNAL only when the caller's perspective differs, per the OpenTelemetry database span conventions. Copy that discipline: an HTTP handler is SERVER, everything it calls is CLIENT.

Rule 3: keep span names low cardinality. Database span naming conventions exist precisely because span names become aggregation keys in your backend; the spec tells instrumentation to use operation names and collection targets rather than raw query text, per the database span conventions. Do the same: vector.search always, never vector.search for "how do refunds work". Put the query in an attribute or event, not the name.

Here is the shape in Python, using only the API in the hot path:

from opentelemetry import trace

tracer = trace.get_tracer("rag.pipeline")

def answer(question: str, index: str, top_k: int, candidates: int) -> str:
    with tracer.start_as_current_span("rag.request") as root:
        root.set_attribute("rag.index", index)
        root.set_attribute("rag.top_k", top_k)

        with tracer.start_as_current_span("embed.query") as s:
            s.set_attribute("gen_ai.operation.name", "embed")
            vector = embed(question)

        with tracer.start_as_current_span("vector.search") as s:
            s.set_attribute("db.collection.name", index)
            s.set_attribute("rag.top_k", top_k)
            hits = index.query(vector, top_k=top_k)
            s.set_attribute("rag.results", len(hits))

        with tracer.start_as_current_span("rerank") as s:
            s.set_attribute("rag.candidates", candidates)
            ranked = rerank(question, hits, top=candidates)
            s.set_attribute("rag.kept", len(ranked))

        with tracer.start_as_current_span("prompt.assemble"):
            prompt = build_prompt(question, ranked)

        with tracer.start_as_current_span("model.generate") as s:
            s.set_attribute("gen_ai.request.model", model_id)
            text = generate(prompt)
            s.set_attribute("gen_ai.usage.output_tokens", count_tokens(text))

    return text

The with blocks are the whole point: each span starts and ends where the work starts and ends, so the timings nest cleanly by construction.

Attributes are where traces stop being pictures and start being data. The question to ask of every attribute is: which decision would I make differently if this value were different? Attributes that fail that test are noise that costs storage.

Two habits separate good attribute design from wishful thinking.

Record counts, not content. rag.results=8 is cheap and queryable. The eight document texts are expensive, may contain user data, and belong in a log line correlated by trace ID. The SigNoz structured logging guide makes the general case well: fields in named keys can be filtered, grouped, and alerted on, while a free-text sentence cannot. The same logic applies one level up, inside spans.

Record the outcome, not just the happy path. Set the span status to error and attach the HTTP status code when a dependency fails. A trace where vector.search failed but rag.request reports success is worse than no trace, because it teaches the reader to distrust the data.

Where the OpenTelemetry semantic conventions already cover your operation, use their keys instead of inventing your own. HTTP attributes like http.request.method and http.response.status_code, and database attributes like db.collection.name, are exactly the sort of "common names for different kinds of operations and data" that the OpenTelemetry concepts index lists as the purpose of semantic conventions. For generative model calls, the gen_ai.* namespaces in the conventions repository are the closest thing to a standard, and adopting them now means your dashboards survive a client library rewrite.

Once traces are flowing, the diagnosis is arithmetic. Take a slow trace and subtract each child's duration from the parent's.

rag.request         ████████████████████████████████████  1,240 ms
embed.query         ██                                        65 ms
vector.search       █████                                     180 ms
rerank              ████████████                              420 ms
prompt.assemble     █                                          12 ms
model.generate      ████████████                              410 ms
  (unaccounted)     █████                                      153 ms

The children sum to 1,087 ms. The parent took 1,240 ms. The 153 ms of unaccounted time is not a mystery: it is your code between the spans (function call overhead, serialization, logging) plus any waiting you did not wrap, such as a queue hop or a cache miss you forgot to instrument.

Now apply the decision rule that separates the four suspects:

• Rerank and generation dominate, retrieval is fast. You are paying for accuracy. Cut rag.candidates from, say, 50 to 20 and watch recall and latency together, since reranker cost scales with candidate count, per Pinecone.
• vector.search dominates. The index is the suspect. Check whether a filter is being applied, whether top_k ballooned, and whether the database host is under memory pressure, the factors the OneUptime vector monitoring guide lists as the usual causes of query times moving from tens of milliseconds to hundreds.
• embed.query dominates. This is a network and provider problem, not a retrieval problem. Look at p99 rather than mean, because rate limit retries show up in the tail. The novaaiops guide to the golden signals makes the point that averages hide exactly the slow requests users notice, and that failed requests often fail fast, dragging averages down while your error rate climbs.
• Children sum to far less than the parent. The overhead is yours: unawaited futures, synchronous JSON encoding of large payloads, or an uninstrumented hop. This is the case most teams miss, because every instrumented thing looks fine.

A worked example makes the payoff concrete. Suppose a 1,200 ms p95 target. Trace A shows 1,240 ms with rerank at 420 ms and generation at 410 ms; dropping candidates to 20 cuts rerank roughly proportionally to about 170 ms, bringing p95 to about 990 ms with no infrastructure change. Trace B shows 1,240 ms with vector.search at 700 ms and everything else small; no prompt or model change will help, and the fix is an index rebuild or a smaller top_k. Two identical durations, two completely different owners. That is the insight the first search result does not give you, and it only exists if the spans exist.

Overlapping spans. If you wrap a retrying helper in a span and its individual attempts in child spans, the children sum to more than the parent and your subtraction lies. Fix it by making retries child spans and treating the parent as wall clock, then never comparing parent duration to the sum of children without knowing which convention you used. Alternatively, record retry attempts as span events, which are timestamped annotations on one span rather than separate units of work, per the traces documentation.

Missing spans at the seams. The most common missing span is the one around the vector store client's gRPC or HTTP call when the client library does not emit spans itself. If your vector.search span wraps only the local function that builds the request, you are measuring CPU in your own process, not the network. The vector database monitoring guide shows the intended shape: the SDK wraps the client call so timing includes the transport.

Broken context propagation. If the embedding call happens in a worker process or a thread pool and the trace context is not propagated, the child span becomes a second root trace and the parent shows a gap. Context propagation is the mechanism that lets spans be correlated and assembled into a trace regardless of where they are generated, per the traces concept page, and the OpenTelemetry concepts index treats it as its own topic for that reason. The symptom to look for is a suspiciously large unaccounted gap plus a second trace with nearly the same timestamp.

Sampling that hides the tail. Head sampling at a low rate will drop most of your slowest traces if slowness correlates with load. Keep a high sample rate for errors and for traces exceeding a duration threshold, and rely on the parent to carry the decision. The OpenTelemetry concepts index lists sampling as a first-class concern with multiple options for exactly this reason.

Traces without logs. A span tells you vector.search took 700 ms; it does not tell you which filter clause ran. Correlate. Structured logs with a trace_id and span_id field let you pivot from the slow span to the exact log lines, the practice the OpenObserve production logging guide recommends with a shared field schema across services. The IBM observability overview groups logs, metrics, and traces as complementary signals for the same reason: each answers a different question.

Treating traces as the only signal. Traces explain one request. Metrics explain the population. The Google SRE book's guidance on monitoring distributed systems is to pick a small number of signals that surface most problems, and per-stage latency percentiles derived from your spans fit that role. Use traces to explain the outlier and metrics to notice it. The CrowdStrike observability primer and the Elastic blog on the three pillars both make the same division of labor: metrics answer what is happening, logs answer why, traces answer where.

Instrumentation pays off only if it changes what someone does at 2 a.m. The workflow below turns spans into decisions.

Two governance habits keep this honest. First, define the SLI you actually care about, such as the fraction of requests whose full trace completes under a threshold, rather than the fraction under threshold per stage. The OpenObserve guide to setting meaningful SLOs is blunt that SLOs built from infrastructure health rather than user experience are theater, and a per-stage threshold can be met while the end-to-end request fails. Second, review the span tree in code review the way you review database schema changes. Adding a stage to the pipeline without adding a span is the same class of mistake as adding a query without an index.

If your model calls span several providers and you want one trace tree rather than three vendor-specific ones, a single OpenAI-compatible endpoint for text, image, video, audio, and embedding models removes a whole class of propagation bugs, and Quanta Inference is built for that role. For teams that want the pipeline, the spans, and the SLO review handled together with someone who has done it before, Quanta takes a described system and pairs you with a specialist to build it.

One last structural note: the OpenTelemetry project overview describes clients as organized into signals that share context propagation but function independently, with API, SDK, and semantic conventions as separate package types. That separation is your insurance policy. Instrument against the API and the conventions, keep the SDK at the edge, and the day you change observability backends, per OpenTelemetry's own description of vendor neutrality, your RAG spans move with you unchanged.