Observability: Logs, Metrics and Traces

The OpenTelemetry primer defines a log as a timestamped message emitted by services or other components, and notes that logs on their own often lack the context to track code execution. Two practices make them far more useful.

Write structured logs. New Relic describes structured logging as recording key-value pairs that provide context about the event instead of raw text. Compare the two forms.

ERROR payment failed for user 812 after 3 retries

{
  "timestamp": "2026-03-04T10:15:22.481Z",
  "level": "error",
  "service": "checkout",
  "message": "payment failed",
  "user_id": "812",
  "retries": 3,
  "provider_status": 502,
  "request_id": "req_7f2c",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

The first is easy to read and hard to search. The second can be filtered by service, counted by provider status, and joined to a trace.

Use the same field names everywhere. OpenObserve's guide lists consistent field names across services as a core property, with a shared set such as level, trace_id, request_id and a non-identifying user_id. If one service logs userId and another logs uid, every cross-service query becomes a puzzle.

A few more rules keep logs useful.

• Make levels mean something. The same guide warns that when every log line is an error, nothing is. Reserve error for failures someone should look at.
• Never log secrets or personal data. Passwords, tokens, card numbers and health data do not belong in logs. Redact at the logger, not in the search tool.
• Log at boundaries. Record one line when a request arrives and one when it completes, with duration and status, plus lines for failures and important decisions.

The OpenTelemetry primer defines metrics as aggregations over a period of time of numeric data about your infrastructure or application, such as error rate, CPU utilization and request rate. Metrics are cheap to store and fast to query, which makes them the basis of dashboards and alerts.

Google's Site Reliability Engineering book names the four that matter most for any user-facing system: latency, traffic, errors, and saturation. If you can only measure four metrics, measure these.

Two details from the same chapter prevent common mistakes.

1. Separate the latency of errors. A request that fails fast because the database connection dropped can make average latency look better while users suffer. Track error latency on its own.
2. Do not trust averages. The book gives the example of a service with an average latency of 100 ms in which 1% of requests might easily take 5 seconds. It recommends collecting request counts in latency buckets, a histogram, so you can see the slow tail. Report percentiles such as p95 and p99, not the mean.

A distributed trace records the path taken by a single request as it moves through services. It is made of spans. Each span represents one unit of work, such as an HTTP call, a database query or a function, with a start time, a duration and attributes. The first span is the root, and child spans nest under it.

When a user loads a page, the request might pass through an API gateway, a backend service and a database. Each step becomes a span, and the trace shows them as a waterfall. A trace answers questions that logs and metrics struggle with: which of the six calls made this request slow, and did the slow part happen in your code or in a dependency.

Tracing depends on context propagation. The first service creates a trace ID and passes it to every downstream call in a header, so every span and log line can carry it. Most instrumentation libraries do this automatically for common HTTP clients and servers.

Traces generate a lot of data. CrowdStrike's overview notes that the sheer volume of data from tracing systems at scale is overwhelming, so traces are usually sampled. A practical approach keeps every trace that contains an error or exceeds a latency threshold, and a small share of the rest.

The signals are most valuable together. Elastic's overview of the three pillars warns that without correlating the data, users face a variety of blind spots. The OpenTelemetry primer makes the same point about logs: they become far more useful when correlated with a trace and a span.

The workflow you want looks like this.

1. A metric shows p99 latency for checkout rising.
2. You filter traces for checkout requests slower than two seconds in that window.
3. One trace shows a payment provider call taking most of the time.
4. You search logs by that trace's trace_id and see the retries and the provider's error code.

Every step uses a shared identifier. To make it possible:

• Add trace_id and span_id to every log line, which most OpenTelemetry logging integrations can do for you.
• Use the same service name in metrics, logs and traces.
• Record exemplars, sample trace IDs attached to metric data points, if your backend supports them, so a click on a latency spike opens a matching trace.

Telemetry tells you what is happening. Service level objectives tell you whether it matters. The OpenTelemetry primer defines an SLI as a measurement of a service's behavior that is best taken from the perspective of users, and an SLO as the way reliability is communicated to the organization, by attaching SLIs to business value.

A practical guide on Medium sets out the three parts with examples: an SLI such as the fraction of requests that succeed, an SLO such as 99.9% availability over 30 days, and an error budget of 0.1% downtime allowed (43 minutes per month). The error budget is a decision tool. While budget remains, the team ships freely. When it is spent, reliability work comes first.

Good SLOs share some traits.

• They measure what users experience, such as successful checkouts or page loads under a threshold, not CPU usage.
• They cover a rolling window, often 28 or 30 days.
• They are below 100%. No system is perfect, and a 100% target leaves no room to ship.

Alerting follows from the SLO. Google's SRE book is blunt about the cost of noise: paging a human is expensive, and when pages occur too often, people second-guess, skim, or even ignore incoming alerts. It recommends paging on symptoms, what is broken from the user's point of view, rather than on every possible cause.

Every page should be actionable, urgent and tied to user impact. Review alerts that fired without needing action and delete or downgrade them.

OpenTelemetry is the common standard for producing all three signals. The project describes itself as an observability framework and toolkit that is open source, as well as vendor- and tool-agnostic, and lists as a benefit that you own the data you generate with no vendor lock-in. You instrument once and can send data to any compatible backend.

A sensible first setup for a small team:

1. Add the OpenTelemetry SDK and automatic instrumentation for your language. This usually traces incoming HTTP requests, outgoing HTTP calls and database queries with little code.
2. Run the OpenTelemetry Collector as a small service that receives telemetry, adds attributes such as environment, samples traces and forwards everything to your backend.
3. Switch your logger to structured JSON and include the trace ID from the current context.
4. Build one dashboard per service with the four golden signals.
5. Define one or two SLOs for the user journeys that matter most, such as sign-in and checkout, and alert on error budget burn.
6. Practice the workflow from metric to trace to log during a calm week, so it is familiar during an incident.

Teams that rely on AI agents in production need the same visibility into model calls, tool calls and costs. Quanta Intelligence agents and Inference requests are easier to debug when each call is logged with the same request and trace IDs as the rest of the system.